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Executive  Summary 


On  August  12,  2002,  the  Assistant  Secretary  of  the  Army  for  Acquisition,  Logistics,  and  Technol¬ 
ogy  (ASA(ALT))  initiated  the  Army’s  Strategic  Software  Improvement  Program  (ASSIP).  The 
purpose  of  the  ASSIP  is  to  improve  the  way  in  which  the  Army  acquires  software-intensive  sys¬ 
tems. 

As  part  of  the  ASSIP,  the  Army  funded  the  Carnegie  Mellon"  Software  Engineering  Institute 
(SEI)  to  conduct  software  architecture  evaluations  on  Army  systems  using  the  SEI  Architecture 
Tradeoff  Analysis  Method®  (ATAM®)  from  2002  through  2007.  The  AT  AM  is  a  method  for  ana¬ 
lyzing  architectures  relative  to  their  quality  attribute  requirements.  AT  AM -based  architecture 
evaluations  identify  architecture  risks,  which  are  potentially  problematic  architectural  decisions 
relative  to  the  system’s  ability  to  satisfy  its  quality  attribute  requirements.  Additionally,  in  cases 
when  a  system’s  architecture  did  not  exist  or  was  too  immature  to  evaluate,  the  ASSIP  sponsored 
SEI  Quality  Attribute  Workshops  (QAWs).  The  QAW  is  a  method  for  eliciting  quality  attribute 
requirements  that  are  essential  for  designing  a  system’s  architecture.  During  this  same  period, 
several  other  Army  programs  funded  their  own  ATAM  evaluations  and  QAWs.  A  total  of  12  Ar¬ 
my  programs  conducted  ATAM  or  QAW  evaluations  during  this  period,  and  all  participated  in 
this  study. 

The  purpose  of  this  report  is  to  convey  the  results  of  a  survey  that  elicited  the  perceived  impact 
the  ATAM  evaluations  and  QAWs  had  on  system  quality  and  the  practices  of  the  acquisition  or¬ 
ganization.  The  survey  was  constructed  to  determine  how  the  programs  were  impacted  in  terms  of 
the  quality  of  the  system;  the  practices  of  the  involved  program  office,  stakeholders,  and  suppli¬ 
ers;  and  the  overall  perceived  value  of  the  ATAM  and/or  QAW  engagements. 

Overall,  the  survey  results  suggest  that  the  Army  programs  received  benefit  from  the  use  of  the 
ATAM  and  QAW,  as  shown  in  the  following  results: 

•  Six  of  the  12  programs  reported  that  it  cost  less  to  use  the  QAW  to  elicit  quality  attribute 
requirements  and  the  ATAM  to  evaluate  their  software  architecture  than  the  techniques  they 
traditionally  have  used.  Moreover,  independent  of  whether  the  programs  reported  less  or 
more  cost,  they  all  reported  results  that  were  at  least  as  good,  and  often  better,  than  the  re¬ 
sults  they  traditionally  obtained. 

•  Ten  of  the  12  programs  that  conducted  an  ATAM  evaluation  and/or  QAW  reported  that  the 
method  provided  an  informed  basis  for  the  program  office  and  the  supplier  to  better  under¬ 
stand  and  control  the  software  development  cost  and  schedule. 

•  All  programs  found  that  using  the  ATAM  and/or  QAW  increased  their  understanding  of  the 
system’s  quality  attribute  requirements,  design  decisions,  and  risks.  This  is  consistent  with 
what  we  heard  when  we  held  an  ASSIP-sponsored  Army  architecture  workshop  in  2007, 
where  the  participants  told  us  that  using  the  ATAM  and/or  QAW  could  be  used  to  reduce 
acquisition  risk  for  the  DoD. 


Carnegie  Mellon,  Architecture  Tradeoff  Analysis  Method,  and  ATAM  are  registered  in  the  U.S.  Patent  and 
Trademark  Office  by  Carnegie  Mellon  University. 
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•  Overall,  the  programs  felt  that  the  use  of  the  AT  AM  and/or  QAW  provided  a  good  mechan¬ 
ism  for  the  program  office,  suppliers,  and  stakeholders  to  communicate  their  needs  and  un¬ 
derstand  how  they  are  met. 

•  A  minority  of  the  respondents  felt  that  using  the  methods  would  result  in  overall  cost  and 
schedule  reductions  for  their  respective  programs.  Further  analysis  revealed  that  the  context 
of  use  had  a  significant  impact  on  this  response.  For  example,  a  lack  of  commitment  (or 
mandate)  to  mitigate  the  risks  found  by  evaluating  the  architecture  would  obviously  limit  the 
ultimate  impact  of  the  evaluation. 

•  A  majority  of  the  respondents  felt  that  using  the  ATAM  and/or  QAW  led  to  an  improved 
architecture  (8  of  12),  and  a  higher  quality  system  (6  of  10).  Again,  contextual  factors  had  a 
significant  impact  on  these  findings,  leading  us  to  believe  that  under  appropriate  acquisition 
conditions  these  practices  are  very  likely  to  have  a  positive  impact  on  system  quality. 

In  summary,  the  data  gathered  for  this  study  confirms  that  the  use  of  AT  AM-based  architecture 
evaluations  and  QAWs  are  generally  beneficial  to  DoD  system  acquisitions  and  suggests  that 
maximal  benefit  is  achievable  only  if  architecture-centric  practices  are  built  into  the  acquisition 
process. 
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Abstract 


The  Army  Strategic  Software  Improvement  Program  (ASSIP)  is  a  multiyear  effort  targeted  at  im¬ 
proving  the  way  in  which  the  Army  acquires  software-intensive  systems.  The  ASSIP  has  funded  a 
number  of  programs,  in  conjunction  with  the  Carnegie  Mellon®  Software  Engineering  Institute 
(SEI),  to  conduct  software  architecture  evaluations  using  the  SEI  Architecture  Tradeoff  Analysis 
Method®  (AT AM®).  Additionally,  in  cases  when  a  system’s  architecture  did  not  exist  or  was  not 
ready  to  evaluate,  the  ASSIP  sponsored  SEI  Quality  Attribute  Workshops  (QAWs).  During  the 
period  of  this  effort,  several  other  programs  funded  their  own  AT  AM  evaluations  and  QAWs.  The 
goal  of  this  study  was  to  determine  the  benefits  associated  with  using  the  AT  AM  and  QAW. 

This  special  report  describes  the  results  of  a  study  of  the  impact  that  the  AT  AM  evaluations  and 
QAWs  had  on  Army  programs.  All  12  programs  that  used  the  AT  AM  and/or  QAW  responded  to  a 
questionnaire  whose  objective  was  to  determine  the  impact  of  the  experience  in  terms  of  the  quali¬ 
ty  of  the  system,  the  practices  of  the  involved  program  office,  stakeholders,  and  suppliers,  and  the 
overall  value  of  the  engagement. 

The  data  gathered  confirms  that  the  use  of  AT  AM-based  architecture  evaluations  and  QAWs  are 
generally  beneficial  to  system  acquisitions  and  suggests  that  maximal  benefit  is  achievable  only  if 
architecture-centric  practices  are  built  into  the  acquisition  process. 


Carnegie  Mellon,  Architecture  Tradeoff  Analysis  Method,  and  ATAM  are  registered  in  the  U.S.  Patent  and 
Trademark  Office  by  Carnegie  Mellon  University. 
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1  Introduction 


On  August  12,  2002,  the  Assistant  Secretary  of  the  Army  for  Acquisition,  Logistics,  and  Technol¬ 
ogy  (ASA(ALT))  initiated  the  Army’s  Strategic  Software  Improvement  Program  (ASSIP).  The 
purpose  of  the  ASSIP  is  to  improve  the  way  in  which  the  Army  acquires  software-intensive  sys¬ 
tems.  The  ASSIP  is  predicated  on  the  idea  that  better  acquisition  practices  (such  as  rigorous  eval¬ 
uation  of  software  architectures  developed  for  the  systems  being  acquired)  will  lead  to  better  sys¬ 
tems  and  overall  results  [Blanchette  2007]. 

As  part  of  the  ASSIP,  the  Army  funded  the  Carnegie  Mellon  '  Software  Engineering  Institute 
(SEI)  to  conduct  software  architecture  evaluations  on  nine  Army  systems  using  the  SEI  Architec¬ 
ture  Tradeoff  Analysis  Method"  (ATAM®).  The  ATAM  is  a  method  for  evaluating  architectures 
relative  to  their  quality  attribute* 1  requirements.  ATAM-based  architecture  evaluations  identify 
architecture  risks,  which  are  potentially  problematic  architectural  decisions  relative  to  the  sys¬ 
tem’s  ability  to  satisfy  its  quality  attribute  requirements.  System  development  suppliers,2  as  the 
architects  of  their  respective  systems,  participated  in  the  ATAM  evaluations  by  presenting  their 
architectures  to  the  evaluation  teams,  describing  how  the  architecture  satisfies  each  of  the  quality 
attribute  scenarios,  listening  and  responding  to  the  teams’  findings,  and,  ultimately,  by  taking  ap¬ 
propriate  actions  based  on  those  findings. 

Additionally,  in  cases  when  a  system’s  architecture  did  not  exist  or  was  too  immature  to  evaluate, 
the  ASSIP  sponsored  a  companion  method — the  SEI  Quality  Attribute  Workshop  (QAW) — to 
elicit  quality  attribute  requirements.  During  this  same  period,  several  other  Army  programs 
funded  their  own  ATAM  evaluations  and  QAWs.  A  total  of  12  Army  programs  conducted  ATAM 
or  QAW  evaluations  during  this  period,  and  all  participated  in  this  study. 

In  addition  to  funding  these  architecture-related  engagements,  the  ASSIP  also  funded  an  Army 
Software  Architecture  Workshop  that  the  SEI  co-hosted  in  Pittsburgh,  PA  during  May  of  2007. 
The  goal  of  the  workshop  was  to  provide  a  forum  for  participants  to  share  lessons  learned  in  using 
the  ATAM  and  QAW  and  to  examine  enablers  and  barriers  to  adoption.  Participants  clearly  indi¬ 
cated  that  they  found  their  experiences  with  ATAM/QAW  techniques  useful.  Some  of  the  benefits 
cited  were 

•  explicit  capture  of  the  programs’  business  and  mission  goals  and  desired  system  quality 
attributes 

•  rigorous  specification  and  prioritization  of  the  system’s  desired  system  quality  attributes  in 
an  unambiguous  and  testable  form 

•  improved  (or  first  ever)  software  architecture  documentation 


Carnegie  Mellon,  Architecture  Tradeoff  Analysis  Method,  and  ATAM  are  registered  in  the  U.S.  Patent  and 
Trademark  Office  by  Carnegie  Mellon  University. 

Quality  attribute  requirements  are  also  referred  to  as  nonfunctional  requirements. 

Note  that  the  term  supplier  refers  to  the  organization  responsible  for  supplying  the  software,  which  could  be  a 
contractor,  subcontractors,  software  engineering  center  (SEC),  or  other  software  development  organization. 


1  |  CMU/SEI-2009-SR-007 


•  discovery  of  software  architecture  risks  and  overarching  risk  themes  that  were  previously 
unknown  and  which  traditionally  do  not  play  a  role  in  a  preliminary  design  review  (PDR) 

Y et,  for  all  the  positive  feedback,  the  workshop  could  not  answer,  in  any  quantitative  manner, 
challenging  questions  about  the  value  of  ATAM/QAW  practices.  Indeed,  one  of  the  recommenda¬ 
tions  was  to  develop  a  good  case  study  that  includes  return  on  investment  (ROI)  data,  with  the 
expectation  that  such  a  study  would  provide  the  necessary  impetus  for  broader,  more  consistent 
adoption  of  software  architecture  practices  across  Army  acquisition  programs. 

The  difficulty  in  determining  the  ROI  for  ATAM/QAW  practices  stems  from  two  facts:  (1)  DoD 
programs  do  not  traditionally  obtain  detailed  software  cost  data  on  development  contracts  and  (2) 
the  primary  payoff  of  conducting  an  architecture  evaluation  is  cost  avoidance  due  to  mitigating 
risks  that  would  otherwise  require  costly  rework  downstream.  The  study  discussed  in  this  report  is 
a  first  attempt  at  determining  the  ROI  of  ATAM/QAW  practices  in  a  more  systematic  manner. 

The  study  demonstrates  value  in  taking  steps  towards  a  more  disciplined  approach  to  measuring 
results  by  moving  from  measuring  reactions  and  perceived  value  to  measuring  what  participants 
learned  during  the  ATAM/QAW,  what  knowledge  and  skills  they  applied  after  the  engagement, 
and  what  the  consequences  were  in  terms  of  impact  on  programmatics,  product,  and  practices. 

1.1  Objective  of  This  Study 

The  objective  of  the  study  documented  in  this  report  was  to  determine  the  value  the  Army  pro¬ 
grams  received  from  using  the  ATAM  and  QAW.  ASA(ALT),  in  particular,  desired  evidence  that 
the  Army  programs  had  saved  money,  seen  substantial  changes  or  improvements,  and  overall  had 
benefitted  from  a  positive  impact  of  the  AS  SIP-funded  effort  in  architecture  evaluation.  This  im¬ 
pact  data  would  enable  the  Army  decide  whether  these  practices  should  be  considered  for  broad 
adoption  across  the  Army.  Accordingly,  the  SEI  was  tasked  by  ASA(ALT)  to  determine  and  re¬ 
port  on  the  impact  of  these  architecture  practices  and  lessons  learned  on  Army  programs.  Table  1 
identifies  the  Army  programs  and  the  architecture-related  practices  they  were  surveyed  about  in 
this  study. 

Table  1:  Participating  Army  Programs  and  the  Architecture-Related  Practices  Employed 


Army  Programs  (in  alphabetical  order) 

ATAM 

QAW 

Aerial  Common  Sensor  (ACS) 

✓ 

✓ 

Army  Battle  Command  System  (ABCS) 

✓ 

Command  Post  of  the  Future  (CPoF) 

✓ 

Common  Avionics  Architecture  System  (CAAS) 

✓ 

Distributed  Common  Ground  Station  -  Army  (DCGS-A) 

✓ 

V 

Force  XXI  Battle  Command,  Brigade-and-Below  (FBCB2) 

✓ 

Future  Combat  Systems  (FCS) 

✓ 

✓ 

Integrated  Fired  Control  (IFC) 

✓ 

✓ 

Joint  Tactical  Common  Operational  Picture  Workstation  (JTCW) 

✓ 

Manned/Unmanned  Common  Architecture  Program  (MCAP) 

✓ 

One  Semi-Automated  Forces  (OneSAF) 

✓ 

Warfighter  Information  Network  -  Tactical  (WIN-T) 

✓ 
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Collectively,  these  12  Army  programs  conducted  1 1  ATAM  architecture  evaluations  and  5  QAWs 
from  2002  through  2007. 

1.2  Basis  for  Evaluating  the  Impact  of  ATAM  and  QAW  Architecture  Practices 

A  questionnaire  was  developed  to  survey  the  experiences  of  the  12  Army  programs  that  partici¬ 
pated  in  the  ATAM/QAW  engagements.  The  questionnaire  was  designed  to  elicit  information  on 
the  impact  of  using  the  ATAM  and  QAW  on  quality  improvements  that  were  realized  by  the  Ar¬ 
my  programs.  The  quality  improvements  of  interest  were 

•  programmatic  improvements  (cost  and  schedule  aspects) 

•  product  improvements  (system  and  software  development) 

•  practice  improvements  (acquisition  and  development  practices) 

These  improvements  were  considered  from  the  standpoint  of  when  they  were  realized — that  is, 
whether  they  occurred  during  the  preparation  and  execution  of  the  ATAM  and  QAW  or  after¬ 
wards.  Table  2  depicts  the  matrix  that  resulted  from  considering  the  three  noted  quality  improve¬ 
ments  of  interest  with  respect  to  when  they  were  realized.  Each  cell  of  the  matrix  was  then  used  to 
distill  the  topics  for  the  questionnaire.* * 3 

Table  2:  Criteria  for  Evaluating  the  Impact  of  Army  AT  AMs  and  Q/AWs 


Quality 

Improve¬ 

ments 

Preparation  and  Execution 

Post-Engagement  Activities 

Programmatic 

(Cost  and 
Schedule) 

Effective  in  terms  of 

•  cost 

•  effort 

Improved 

•  program  schedule  performance 

•  program  cost  performance 

Product 

(System  and 
Software) 

Clarification,  discovery,  and  use  of 

•  quality  attribute  requirements 

•  architecture  documentation 

•  risks  and  risk  themes 

Improved 

•  software  architecture 

•  system  qualities/capabilities 

•  warfighter  effectiveness 

Practices 

(Acquisition  and 
Development) 

Foster  communication  among  program  office, 
suppliers,  and  stakeholders  to 

•  understand  and  control  cost  and  schedule 

•  communicate  quality  attribute  requirements 

•  evaluate  the  architecture 

•  improve  the  architecture 

Organizational  changes 

•  use  of  the  results 

•  use  of  the  practices  in  the  short  term 

•  adoption  of  the  concepts 

•  adoption  of  the  methods 

•  training  personnel  to  conduct  the  methods 

Programmatic  considerations  for  preparation  and  execution  led  to  questions  that  addressed  cost 
and  effort  aspects  of  an  ATAM  evaluation  or  QAW  as  compared  to  how  the  software  architecture 
is  typically  evaluated  and  requirements  are  typically  elicited.  Considering  post  activities  led  to 
questions  that  addressed  long-term  improvements  in  cost  and  schedule  performance. 

Product  considerations  for  preparation  and  execution  led  to  questions  that  addressed  the  clarifica¬ 
tion,  discovery,  and  use  of  quality  attribute  requirements,  architecture  documentation,  and  archi- 

3  Note  that  one  questionnaire  was  developed  for  both  the  ATAM  and  QAW  although  only  a  subset  of  the  ques¬ 

tions  was  relevant  for  the  QAW. 


3  |  CMU/SEI-2009-SR-007 


tecture  risks.  Considering  post  activities  led  to  questions  that  addressed  the  long-term  improve¬ 
ments  in  system  quality  and  warfighter  effectiveness. 

Practice  considerations  for  preparation  and  execution  led  to  questions  that  addressed  communica¬ 
tion  among  the  stakeholders  and  use  of  the  results  of  the  engagement  as  an  informed  basis  on 
which  to  improve  acquisition  practices  (especially  those  that  involved  communication  between 
the  program  and  the  supplier).  Considering  follow-on  activities  led  to  questions  that  addressed 
changes  in  the  behavior  of  the  organization  in  both  the  short  term  (e.g.,  use  of  the  results,  use  of 
practices  to  gather  additional  results)  and  in  the  longer  tenn  (e.g.,  adopting  new  practices  and  in¬ 
vesting  in  training). 

1.3  The  Program  Impact  Questionnaire 

The  quality  improvements  of  interest  were  used  as  the  basis  for  creating  a  questionnaire  to  elicit 
the  perceived  impact  the  ATAM  evaluations  and  QAWs  had  on  system  quality  and  the  practices 
of  the  acquisition  organization. 

The  survey  was  constructed  to  determine  how  the  programs  were  impacted  in  terms  of  quality  and 
cost;  whether  there  were  follow-on  (i.e.,  post-ATAM/QAW)  activities,  whether  the  QAW  or 
ATAM  was  subsequently  adopted  as  a  future  program  practice,  and  what  the  overall  value  of  the 
ATAM/QAW  engagements  were  perceived  to  be. 

The  questionnaire  was  organized  into  four  sections: 

1 .  Conducting  the  ATAM/QAW — elicited  information  about  product  and  practice  improve¬ 
ments  during  preparation  and  execution  of  the  method 

2.  Follow-On  ATAM/QAW  Activities — elicited  information  about  practice  improvements  dur¬ 
ing  the  post  activities,  focusing  on  how  the  engagement  affected  the  immediate  behavior  of 
the  organization 

3.  Adoption  of  ATAM/QAW — elicited  information  about  practice  improvements  during  the 
post  activities,  focusing  on  how  the  engagement  affected  the  long-term  acquisition  practices 
leading  to  adoption  of  ATAM  and/or  QAW  as  part  of  acquisition  practices. 

4.  Overall  Impact  — elicited  information  about  short-term  and  long-term  programmatic  im¬ 
provements  and  long-term  product  improvements;  in  addition,  it  provided  survey  respondents 
an  opportunity  to  share  comments  on  how  they  perceived  the  overall  impact  of  the  engage¬ 
ment  and  about  the  engagement  in  general. 

Survey  respondents  were  also  asked  to  comment  on  the  circumstances  that  influenced  their  res¬ 
ponses.  This  information  helped  us  understand  the  contextual  factors  that  might  influence  the 
findings  and  the  appropriate  acquisition  conditions  under  which  these  practices  could  be  applied 
to  have  a  greater  impact  on  system  quality. 

We  collected  data  from  each  of  the  participating  programs  with  the  understanding  that  the  results 
would  be  aggregated  across  the  programs,  and  data  specific  to  a  program  would  not  be  revealed. 

1.4  Organization  of  This  Report 

This  special  report  is  organized  as  follows: 
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•  Section  2  presents  the  findings  and  provides  a  measure  of  the  overall  estimated  value  the 
Army  programs  received  from  these  engagements,  in  terms  of  the  quality  improvements  that 
were  realized. 

•  Section  3  describes  the  context  (with  respect  to  planning,  timing,  need,  motivation,  accom¬ 
modation,  and  follow  through)  in  which  the  programs  conducted  the  ATAM/QAW  and  the 
effect  that  context  had  on  benefiting  from  the  results. 

•  Section  4  draws  conclusions  for  incorporating  architecture  practices  in  system  acquisitions 
that  could  aid  Army  programs  in  achieving  maximum  impact. 

The  appendices  include  a  list  of  acronyms  used  in  the  report,  an  overview  of  the  AT  AM  and 
QAW,  the  ATAM/QAW  Impact  Questionnaire  that  was  completed  by  each  of  the  programs,  and 
the  summary  comments  regarding  impact  provided  by  the  respondents. 


5  |  CMU/SEI-2009-SR-007 


2  Impact  of  Army-Sponsored  ATAMs/QAWs 


This  section  considers  the  findings  from  the  perspectives  of  the  quality  improvements  of  interest: 
programmatic,  product,  and  practice  improvements. 

2.1  Programmatic  Improvements  (Cost  and  Schedule  Aspects) 

The  responses  relating  to  conducting  the  ATAM/QAW  engagements  and  the  post-engagement 
activities  provide  different  perspectives  on  impact  with  respect  to  cost  and  schedule. 

Findings  relating  to  conducting  the  ATAM/QAW 

We  asked  respondents  to  compare  the  cost  (in  terms  of  up-front  expense  and  effort)  of  conducting 
the  ATAM/QAW  to  the  means,  if  any,  they  would  otherwise  have  used  to  elicit  and  specify  quali¬ 
ty  attribute  requirements  and  evaluate  the  software  architecture.  To  make  a  fair  comparison,  it  is 
important  to  consider  the  level  of  quality  being  produced  for  the  given  cost.  So  we  asked  respon¬ 
dents  to  make  comparisons  from  two  points  of  view:  (1)  compare  the  cost  of  applying  the  alterna¬ 
tive  means  to  produce  results  of  the  same  level  of  quality  as  the  ATAM/QAW,  and  (2)  compare 
the  quality  of  the  results  (e.g.,  quality  attribute  scenarios,  architecture  documentation,  and  archi¬ 
tecture  risks)  of  the  alternative  means  as  they  are  traditionally  used  with  those  of  the 
ATAM/QAW. 

The  results  can  be  arranged  in  the  four  quadrants  as  seen  in  Table  3,  which  shows  how  the  cost 
and  quality  of  the  ATAM/QAW  are  perceived,  compared  to  the  other  techniques  traditionally  em¬ 
ployed  by  the  acquisition  organizations.  (One  program  did  not  respond  to  the  questions  for  this 
category.) 

Table  3:  Quality  of  ATAM/QAW  Output  vs.  Other  Techniques  (shown  as  number  of  programs) 


Conducting  the  method 

Better  (or  same) 

Lesser 

Cheaper  (or  same) 

6 

1 

Costlier 

4 

0 

As  seen  from  the  data  in  the  column  headed  Better  (or  same)  in  Table  3,  10  programs  reported 
that  the  ATAM/QAW  produced  results  that  were  the  same  or  better  quality.  The  one  program  that 
reported  lesser  quality  did  so  for  the  following  reason:  “The  AT  AM  is  a  coarse  grained  (time- 
constrained)  process  for  evaluating  architecture  not  suited  for  precise  analysis  and  delving  deep 
into  details  of  architecture  as  some  other  means  do  to  produce  quality  products.”  However,  that 
program  also  indicated  that  the  ATAM/QAW  involves  less  cost  and  effort  than  traditional  means 
used  to  achieve  the  same  level  of  quality  and  acknowledged  that  the  “QAW  and  ATAM  provided 
benefits  deemed  substantial  enough  to  warrant  adoption  for  future  contracts.” 

Also,  as  shown  in  the  row  labeled  Cheaper  (or  same)  in  Table  3,  seven  programs  reported  that  the 
ATAM/QAW  engagement  was  the  same  or  less  cost  than  their  other  means  to  elicit  and  specify 
quality  attribute  requirements  or  evaluate  architecture  design. 
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Of  the  four  programs  that  reported  more  cost  (see  the  row  labeled  Costlier),  two  responded  that 
the  quality  of  the  results  was  higher  and  they  did  not  have  other  means  of  analysis;  they  indicated 
that  they  would  use  ATAM/QAW  methods  again: 

•  “AT AM  will  again  be  used  for  architecture  evaluation.” 

•  “If  anything,  we  should  have  done  more  of  them  [ATAMs  and  QAWs]  to  continually  recon¬ 
firm  nonfunctional  requirements,  update  the  architecture,  and  get  buy-in  to  the  architecture.” 

The  other  two  indicated  that  the  quality  of  the  results  was  the  same  as  that  for  other  means  of 
analysis  and  offered  observations  related  to  the  timing  of  the  engagement  for  why  a  greater  bene¬ 
fit  was  not  achieved: 

•  “Unfortunately,  due  to  the  breadth  of  impact  of  the  architecture,  the  results  were  not  amena¬ 
ble  to  implementation  except  at  great  cost  and  time.  ATAMs  should  be  conducted  at  the  init¬ 
iation  of  programs  where  architectural  change  can  be  readily  integrated  into  the  program  de¬ 
velopment.” 

•  “The  benefit  of  the  AT  AM  was  equal  to  its  cost.  The  AT  AM  was  conducted  too  soon  in  the 
system  life  cycle  .  .  .  The  AT  AM  and  QAW  has  a  positive  impact  to  the  DoDAF  architec¬ 
ture  construction.” 

Interpretation  of  the  Findings 

Overall,  the  data  show  that  the  ATAM  and  QAW  are  effective  techniques  for  eliciting  quality 
attribute  requirements  and  analyzing  software  architectures;  in  some  cases,  they  are  more  cost- 
effective  than  traditional  analysis  methods. 

This  observation  echoes  the  findings  of  the  2007  Army  Architecture  Workshop.  There,  work¬ 
shop  participants  concluded  that  the  ATAM  offers  a  practical  way  of  evaluating  a  system 
against  its  quality  attribute  requirements  that  typically  take  “a  back  seat”  to  the  functional  re¬ 
quirements.  That  is,  traditionally  acquisition  programs  focus  on  functional  requirements  (at  the 
expense  of  the  quality  attribute  requirements)  to  the  detriment  of  the  fielded  system’s  long-term 
acceptability. 


Findings  relating  to  post-engagement  activities 

Looking  at  the  responses  related  to  post-engagement  actions  provides  another  perspective  on  im¬ 
pact  with  respect  to  cost  and  schedule.  Ten  programs  acknowledged  that  the  ATAM/QAW  me¬ 
thods  would  be  helpful  in  providing  “an  informed  basis  for  the  program  office  and  the  supplier  to 
better  understand  and  control  the  software  development  cost  and  schedule.”  The  two  that  did  not 
offered  these  explanations: 

•  “The  software  architecture  was  not  evaluated.”  (This  respondent  viewed  the  evaluation  as 
being  conducted  on  the  “system  architecture.”) 

•  “Being  existent  on  multiple  platforms,  [the  architecture]  is  not  open  to  change  except  at  great 
cost  and  schedule.” 

Four  respondents  indicated  that  the  use  of  the  methods  did  contribute  to  longer  term  cost  and 
schedule  improvement.  (Three  programs  did  not  respond  to  the  questions  for  this  category.)  Three 
of  the  five  programs  that  did  not  respond  favorably  in  this  category  offered  these  explanations: 
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•  “Since  the  ATAM  results  were  not  substantially  used,  no  impact  was  experienced.” 

•  “The  results  of  the  QAW  and  ATAM  were  not  considered  factors  for  the  .  .  .  acquisition  pro¬ 
gram  baseline.” 

•  “Because  the  ATAM  was  done  late  in  the  development  process  .  .  .  the  ATAM  mostly  vali¬ 
dated  the. .  .architecture  and  approach.” 

These  programs  conducted  their  QAW/ATAM  engagements  late  in  the  development  cycle,  which 
limited  their  flexibility  in  reacting  to  the  findings.  Another  program  reported  that  “these  benefits 
were  still  not  realized  because  the  contractor  has  never  really  embraced  the  [concepts].”  In  this 
case,  the  development  contractor  has  not  committed  in  equal  measure  with  the  project  manage¬ 
ment  office  (PMO)  to  using  the  ATAM/QAW  techniques,  thus  limiting  their  effective  application. 
The  final  program  responding  unfavorably  on  this  point  reported  that  “this  was  a  developmental 
system  under  examination,  the  project  was  not  under  pressure  to  exit  with  ‘fieldable’  software.”  In 
this  case,  the  program  was  conducting  a  research  and  development  project  that  was  close  to  its 
demonstration  phase. 

Interpretation  of  the  Findings 

While  there  is  potential  for  long-term  cost  and  schedule  benefits  from  employing 
ATAM/QAW,  circumstances  can  prevent  realization  of  that  value.  In  any  case,  in  order  to 
achieve  maximum  value,  the  results  of  the  evaluation  must  be  accepted  and  acted  upon,  and 
both  PMO  and  contractor  must  agree  as  to  the  follow  on  actions  to  be  taken.  This  acceptance 
and  agreement  are  only  likely  to  happen  when  a  proactive  approach  is  taken,  and  compliance  is 
contractually  required. 

2.2  Product  Improvements  (System  and  Software  Development) 

The  responses  relating  to  conducting  the  ATAM/QAW  engagements  and  the  post-engagement 
activities  provide  different  perspectives  on  impact  with  respect  to  system  and  software  develop¬ 
ment. 

Findings  relating  to  conducting  the  ATAM/QAW 

The  responses  to  product  improvement  questions  related  to  conducting  the  ATAM/QAW  showed 
that  all  programs  acknowledged  favorable  impact  of  the  ATAM/QAW  techniques  on  their  under¬ 
standing  of  three  key  development  artifacts:  the  quality  attributes,  architecture,  and  risks. 

Overall,  four  programs  reported  moderate  results  in  at  least  one  category;  one  program  reported 
moderate  results  across  the  three  artifacts;  two  reported  moderate  results  in  two  artifacts  and  sig¬ 
nificant  results  in  the  other  one;  one  reported  moderate  results  in  one  artifact  and  significant  re¬ 
sults  in  the  other  two. 

As  Figure  1  shows,  each  program  reported  at  least  moderate  improvements  in  all  three  artifacts. 
The  majority  of  programs,  in  fact,  rated  results  to  be  significant  in  each  of  the  artifacts.  One  pro¬ 
gram  reported  very  substantial  results  with  respect  to  architecture,  and  another  for  risks. 
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Minimal  Moderate  Significant  Very  Substantial 

■  Quality  Attributes  ■  Architecture  ■  Risks 


Figure  1:  Architecturally  Significant  Artifacts  Enhanced  by  A  TAM/QA  W 

Interpretation  of  the  Findings 

These  results  demonstrate  that  the  architecture  team  is  able  to  use  ATAM/QAW  to  achieve  an 
understanding  of  stakeholder  expectations  for  the  system  and  the  implications  of  architectural 
decisions  on  user  needs.  Moreover,  ATAM/QAW  provides  the  architecture  team  and  system 
stakeholders  with  the  means  to  develop  a  joint  understanding  of  the  relevant  risks  to  success. 


Findings  relating  to  post-engagement  activities 

Ultimately,  the  goal  of  ATAM/QAW  is  to  achieve  a  better  product.  The  responses  related  to  post¬ 
engagement  product  improvements  provide  another  perspective  on  impact.  Eight  programs 
acknowledged  that  the  ATAM/QAW  “improved  the  architecture.”  Six  programs  indicated  the  use 
of  the  methods  contributed  to  the  overall  system  quality  and/or  warfighter  effectiveness.  (Two 
programs  did  not  respond  to  the  questions  for  this  category).  The  programs  that  reported  minimal 
results  are  a  subset  of  those  that  reported  minimal  results  for  longer-term  cost  and  schedule  im¬ 
provement;  the  explanations  for  those  results  are  reported  in  Section  2.1. 

Interpretation  of  the  Findings 

The  ATAM  and  QAW  have  yielded  tangible  benefits  for  Army  programs,  including  clarified 
quality  attribute  requirements,  improved  architecture  documentation,  and  reduced  software  de¬ 
sign  risk.  All  of  these  benefits  contribute  to  a  tangible  result:  better  quality  products  for  the 
warfighter.  These  observations  are  consistent  with  the  findings  of  the  2007  Army  architecture 
workshop,  which  noted  the  ability  of  the  ATAM/QAW  to  reduce  software  acquisition  risk,  es¬ 
pecially  in  the  DoD  environment.  As  one  workshop  participant  stated,  “It  was  just  an  excellent 
mechanism  for  getting  a  team  to  work  together  to  discover  risks  and  inconsistencies  that  would 
otherwise  go  undetected.” 
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2.3  Practice  Improvements  (Acquisition  and  Development  Practices) 


The  responses  relating  to  conducting  the  ATAM/QAW  engagements  and  the  post-engagement 
activities  provide  different  perspectives  on  impact  with  respect  to  acquisition  and  development 
practices. 

Findings  relating  to  conducting  the  ATAM/QAW 

The  responses  to  practice  improvement  questions  related  to  conducting  the  ATAM/QAW  showed 
that  the  value  of  the  ATAM/QAW  in  fostering  communication  among  stakeholders  was  rated 
highly.  The  programs  also  acknowledged  that  the  ATAM/QAW  provided  an  informed  basis  for 
the  program  office,  suppliers,  and  stakeholders  to  communicate  their  needs  and  understand  how 
they  are  met. 

As  Figure  2  shows,  the  majority  of  programs  reported  very  substantial  enhancement  of  communi¬ 
cation,  while  several  others  reported  significant  results.  The  one  program  that  reported  minimal 
results  offered  this  explanation:  “It  appeared  to  stakeholders  that  these  activities  did  not  warrant 
action  since  neither  the  time  or  funds  were  readily  available  to  support  them.” 
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Minimal  Moderate  Significant  Very  Substantial 


Figure  2:  Communication  Enhanced  by  A  TAM/QA  W 

Interpretation  of  the  Findings 

The  significance  of  these  results  is  that  stakeholders,  collectively,  are  able  to  use  ATAM/QAW 
to  achieve  a  common  understanding  of  the  system  under  development,  making  it  more  likely 
that  the  completed  product  will  address  stakeholder  expectations  and  user  needs,  thereby  im¬ 
proving  chances  for  program  success. 

Findings  relating  to  post-engagement  activities 

Improving  practices  that  support  development  of  a  better  product  is  a  goal  of  the  ATAM/QAW. 
Thus,  a  final  measure  of  the  impact  of  these  techniques  is  the  extent  to  which  programs 
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incorporate,  or  transition,  them  into  their  own  acqusition  and  development  practices.  The  res¬ 
ponses  related  to  post-engagement  practice  improvements  can  be  grouped  into  five  categories, 
each  building  on  the  next  from  a  basic  level  (numbered  as  “1”)  up  through  increasing  levels  of 
sophistication: 

1.  All  programs  reported  some  use  of  the  artifacts  produced  by  the  ATAM/QAW.  For 

example,  some  put  the  quality  attribute  scenarios  they  developed  into  a  requirements  tracking 
system,  others  improved  their  architecture  documentation,  and  others  formally  tracked  risks 
discovered  durng  the  evaluation. 

2.  Eleven  programs  reported  using  the  techniques  of  the  ATAM/QAW  methods  to  uncover 
additional  risks  by,  for  example,  refining  or  analyzing  additional  scenarios. 

3.  Nine  programs  reported  adopting  the  concepts  of  quality  attribute  requirements  elicitation 
and  architecture  evaluation. 

4.  Seven  programs  reported  adopting  the  ATAM/QAW  methods,  (i.e.,  by  using  or 
specifying  the  use  of  the  practices). 

5.  Three  programs  reported  investment  in  formal  ATAM  evaluation  training. 

Categories  1  and  2  represent  limited  transition,  in  that  investment  in  the  techniques  occurred  in 
the  short  term  to  achieve  an  immediate  impact.  Categories  3,  4,  and  5  represent  more  sustained 
investment  in  adoption  and  training  for  a  longer  term  impact. 

Furthermore,  since  organizations  are  willing  to  invest  time  and  effort,  transition  indicates  the 
value  that  they  place  on  the  ATAM/QAW.  One  respondent  commented,  “The  practices  were 
adopted  because  they  provided  a  well-defined  process  for  architecture  evaluation.”  Another 
offered  this  comment: 

The  ATAM  process  provides  an  excellent  disciplined  approach  for  identifying  driving  quality 
attributes,  associating  those  attributes  with  specific  use  cases,  and  then  describing  specific 
architectural  patterns  that  support,  restrict,  or  are  in  contention  with  other  quality 
attributes.  For  a  software  intensive  system  the  ATAM/QAW  process  would  be  very  beneficial 
at  the  start  of  the  program.  For  programs  already  underway  the  ATAM/QA  W  provides  an 
excellent  opportunity  to  assess  and  strengthen  or  correct  architectural  products  and  process 
through  structured  risk  identification  and  acknowledgement  process. 

Interpretation  of  the  Findings 

These  observations  are  consistent  with  the  findings  of  the  2007  Army  architecture  workshop, 
which  noted  that  the  ATAM  and  QAW  give  stakeholders  “a  voice”  in  the  development  process, 
especially  in  the  DoD  environment,  and  that  the  Army  and  its  support  contractors  have  been 
receptive  to  using  the  QAW  and  ATAM,  demonstrating  that  the  methods  can  be  performed 
collaboratively  without  creating  an  adversarial  environment.  Workshop  participants  concluded 
that  standardizing  the  use  of  methods  such  as  the  ATAM  and  QAW  makes  training  easily 
transferable  from  one  project  to  another  and  provides  a  practical  means  for  capturing  lessons 
learned  and  improving  Army  management  and  conduction  of  its  software-intensive  system  ac¬ 
quisitions. 
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3  Factors  Affecting  Impact 


The  questionnaire  asked  programs  to  comment  on  the  activities  they  did  or  did  not  perform  or 
plan  to  adopt.  Respondents  were  asked  why  they  did  or  did  not  perform  the  activities  and  what 
factors  contributed  to  the  success  of  performing  the  activities  or  were  obstacles  that  hindered  them 
from  doing  so. 

In  the  course  of  studying  and  analyzing  the  impact,  it  became  apparent  that  the  context  in  which 
the  methods  were  used  had  a  significant  impact  on  responses.  A  number  of  factors  played  a  major 
role  in  detennining  the  extent  to  which  the  acquisition  organizations  benefited  from  the  architec¬ 
ture  practices.  The  six  contextual  factors  that  played  a  significant  role  involved  the  planning,  ac¬ 
commodation,  and  timing  of  these  engagements  and  the  need,  motivation,  and  follow  through  for 
these  engagements. 

We  rated  each  of  the  programs  for  each  factor  along  a  3-point  scale,  with  1  indicating  an  undesir¬ 
able  context,  3  indicating  a  desirable  context,  and  2  indicating  something  in  between.  Table  4 
shows  the  12  programs  and  their  ratings  in  terms  of  the  6  contextual  factors. 

Table  4:  Overview  of  Contextual  Factors  Affecting  the  Impact  of  Army  A  TAMs/QA  Ws4 


Program 

Planning 

Timing 

Need 

Motivation 

Accommodation 

Follow 

Through 

1 

1 

1 

1 

1 

1 

1 

2 

1 

1 

2 

1 

1 

1 

3 

1 

1 

2 

1 

1 

2 

4 

2 

1 

2 

2 

1 

2 

5 

2 

2 

2 

2 

2 

2 

6 

1 

2 

3 

3 

2 

2 

7 

1 

2 

3 

3 

2 

3 

8 

1 

3 

2 

3 

2 

3 

9 

2 

3 

3 

3 

2 

2 

10 

1 

1 

3 

3 

3 

3 

11 

1 

3 

3 

3 

3 

3 

12 

3 

3 

3 

3 

3 

3 

The  factors  are  defined  in  the  sections  that  follow,  together  with  an  interpretation  of  the  results. 
While  the  number  of  programs  provides  too  small  a  data  sample  to  show  a  statistical  correlation, 
we  see  evidence  that  a  program  with  a  more  desirable  context  experiences  higher  value  from  their 
ATAM/QAW  experience  than  a  program  with  overall  lower  ratings. 

3.1  Planning  and  Accommodation 

Planning  and  accommodation  have  to  do  with  the  contractual  context.  In  general,  if  the 
ATAM/QAW  is  not  written  into  the  original  request  for  proposal  (RFP)  or  contract  then  the  plan- 


The  programs  are  listed  according  to  their  contextual  factor  rankings  and  do  not  correspond  to  the  alphabetical 
listing  that  was  given  earlier. 
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ning  for  it  is  “reactive”;  the  program  office  typically  must  consider  contract  modifications  or 
scope  adjustments,  often  at  additional  costs.  The  reactive  case  is  therefore  less  desirable  than  the 
proactive  case,  where  the  ATAM/QAW  is  considered  early  in  the  acquisition  cycle.  Proactive 
architecture  evaluations  are  preplanned  and  integrated  into  the  RFP  and  contract  as  an  integral 
part  of  the  acquisition  planning  effort.  The  benefits  of  a  proactive  approach  are  that  all  affected 
parties  are  in  agreement  from  the  outset  of  the  RFP/contract  and  the  cost  and  schedule  of  the  eval¬ 
uation  are  known  entities  that  are  included  as  an  integral  part  of  each  offeror’s  technical  and  cost 
proposals. 

The  contractual  context  for  the  majority  of  the  engagements  discussed  in  this  report  was  reactive; 
with  only  one  being  proactive.  This  is  understandable,  given  that  these  engagements  occurred  ear¬ 
ly  in  the  AS  SIP  program  with  programs  that  were  already  underway.  The  consequence  of  execut¬ 
ing  the  ATAM/QAW  reactively  is  that  there  were  impediments  to  achieving  maximal  value. 

Some  of  the  impediments  of  reactive  planning  could  be  overcome  through  accommodation  be¬ 
tween  the  program  office,  stakeholders,  and  the  supplier  by  considering  the  existing  (or  proposed) 
contractual  circumstances  to  ensure  architecture  practices  can  be  appropriately  accommodated 
within  an  acceptable  cost  and  schedule  window.  When  an  ATAM/QAW  is  done  reactively,  signif¬ 
icant  work  is  needed  to  contractually  accommodate  the  ATAM/QAW  or  the  engagement  (includ¬ 
ing  follow  up  activities)  can  become  problematic.  In  such  cases,  a  program  may  encounter  “push 
back”  from  the  system  developer  because  adding  unplanned  activities  and  events  after  the  original 
contract  is  signed  is  an  added  cost  item  that  will  impact  the  overall  schedule.  Introducing  new 
architecture-centric  practices  within  the  confines  of  an  existing  contract  is  more  costly  than  a 
proactive  approach;  for  example,  architecture  documentation  is  usually  deficient  or  may  have  to 
be  created  anew.  Without  a  proactive  approach,  more  effort  will  have  to  be  expended  negotiating 
with  the  system  developer  to  get  the  level  of  cooperation  and  follow  through  that  is  needed. 

Having  a  champion  or  an  experienced  ATAM  evaluator  who  has  previously  coordinated  such  ef¬ 
forts  can  help  alleviate  some  of  these  obstacles.  Arrangements  must  be  made  to  ensure  the  partici¬ 
pation  of  key  stakeholders  and  to  obtain  the  active  cooperation  of  the  system  contractor.  Such  ac¬ 
commodation  involves  contractual  negotiations  to  reach  agreement  on  funding  and  schedule 
allowances.  Even  in  those  cases  where  both  planning  and  accommodation  were  rated  as  undesira¬ 
ble,  once  the  technical  people  could  be  gathered  to  conduct  the  ATAM/QAW,  the  disciplined 
process  provided  a  means  for  the  participants  to  engage  and  obtain  value  during  the  engagement, 
as  indicated  by  the  preparation  and  execution  results  reported  in  Section  2.  However,  these  factors 
were  more  serious  impediments  to  using  the  results,  especially  after  the  engagement,  as  indicated 
by  the  comments: 

•  “Given  where  we  are  at  now  in  not  [following  up  with  some  of  these  practices]  —  chasing 
functionality  in  lieu  of  quality,  we  realize  benefits  of  adopting  in  our  new  contracts.” 

•  “The  nature  of  the  contract  at  the  time  . . .  precluded  [using  the  developed  quality  attribute 
scenarios]  due  to  cost  and  schedule  impacts  to  the  Acquisition  Program  Baseline.”  This 
same  program  also  noted  that  the  architectural  risks  identified  during  the  ATAM  evaluation 
were  not  tracked  or  managed  for  the  same  reason.  Such  statements  illustrate  a  common  para¬ 
dox  in  program  development:  citing  cost  as  the  reason  for  not  taking  problem  avoidance 
measures  even  though  the  result  of  that  inaction  may  be  more  expensive  problem  resolutions 
later  on. 
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•  “Software  architecture  documentation  for  the  candidate  architecture  evaluated  at  the  AT  AM 
had  limited  access  and  was  not  under  the  purview  of  the  PM  [program  manager].  This  was  a 
lesson  learned  that  contributed  to  the  specification  of  specific  contract  deliverables  for  creat¬ 
ing,  delivering,  managing,  and  evolving  software  architecture  documentation  in  a  subsequent 
solicitation.”  Here,  the  PMO  recognized  the  need  to  have  access  to  the  contractor-developed 
architecture  documentation  and  made  the  appropriate  adjustments  for  future  contracts. 

Insufficient  language  in  the  contract  precludes  adding  activities  or  accommodating  other  changes 
to  follow  through  with  the  evaluation  results;  however,  even  in  situations  where  there  is  a  contrac¬ 
tual  obligation  to  use  the  ATAM/QAW  results,  there  can  still  be  a  risk  if  there  is  no  follow 
through  from  the  program  office.  As  one  respondent  reported,  “Contractor  was  contractually  obli¬ 
gated  to  [update  architecture  documentation  based  on  AT  AM  results].  PMO  realized  the  benefits 
but  the  intent  of  the  architecture  was  never  really  followed  or  instantiated  properly  for  products.” 

While  the  preceding  observations  tell  a  cautionary  tale  about  ensuring  commitment  from  suppliers 
and  stakeholders,  the  PMO  does  have  some  obligation  for  accommodation  and  follow  through. 

For  instance,  the  program  office  in  one  case  made  changes  in  their  processes  to  provide  better 
support  for  review  and  management:  “Before  the  AT  AM  these  attributes  had  been  defined  by  the 
contractor  and  were  well  known  by  the  software  development  staff,  but  the  government  project 
office  was  less  familiar  with  them.  So,  we  found  it  logical  to  incorporate  the  scenario  require¬ 
ments  into  our  review  and  management  of  the  effort.” 

3.2  Timing 

Timing — that  is,  when  in  the  system  development  life  cycle  the  ATAM/QAW  are  deployed — can 
significantly  affect  the  technical  outcome  in  terms  of  the  number  and  type  of  risks  that  are  identi¬ 
fied  and  their  timely  and  cost  effective  resolution.  Conducting  an  ATAM/QAW  too  early  or  too 
late  in  the  system  development  life  cycle  (e.g.,  after  the  system  has  already  been  fielded  for  some¬ 
time)  does  not  yield  the  same  benefit  as  conducting  it  at  a  favorable  time,  such  as  in  the  formative 
stages  of  architectural  design.  Another  more  opportune  time  is  when  a  major  upgrade  is  being 
considered,  which  would  warrant  careful  consideration  of  the  potential  impact  of  new  require¬ 
ments  on  the  existing  architectural  design.  Ideally,  an  architecture  evaluation  would  be  conducted 
before  the  system’s  PDR  or,  at  the  very  latest,  before  the  critical  design  review  (CDR)  or  some 
comparable  event. 

The  timing  of  the  engagements  discussed  in  this  report  was  favorable  in  25%  of  the  cases.  Non- 
optimal  timing  is  a  major  reason  that  several  of  the  responding  programs  reported  lackluster  re¬ 
sults.  Nevertheless,  respondents  reported  that  there  was  value  even  though  the  timing  was  not  op¬ 
timal  and  commented  that  there  could  be  even  greater  impact  if  the  timing  is  adjusted. 

•  “The  scenarios  which  were  developed  by  the  architect(s)  themselves  were  more  firmly 
grounded  in  requirements  and  therefore  usable.  The  brainstorming  nature  of  the  AT  AM 
which  relied  on  stakeholders  having  a  firm  understanding  of  the  system  was  not  well-suited 
to  a  program  with  vague  and  emerging  CONOPS  [concept  of  operations]  and  requirements.” 

Here,  the  timing  of  the  ATAM  was  too  early;  although  a  documented  architecture  existed, 
there  was  enough  uncertainty  among  stakeholders  about  the  goals  of  the  program  to  make 
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the  ATAM  seem  less  useful  to  the  architects.  (Note  that  this  apparent  mismatch  between  ar¬ 
chitecture  development  and  program  goals  may,  in  itself,  be  a  useful  revelation.) 

•  “The  ATAM  did  not  have  as  much  impact  because  most  of  the  work  was  done  up  front  in 
our  case  by  the  architecture  team  between  QAW  and  ATAM.” 

•  One  program  acknowledged  the  need  to  use  the  methods,  saying,  “AT AMs  should  be  con¬ 
ducted  at  the  initiation  of  programs  where  architectural  changes  can  be  readily  integrated  in¬ 
to  the  program  development.” 

•  For  one  program,  the  value  of  the  ATAM  evaluation  was  in  demonstrating  the  quality  of  the 
architecture  to  stakeholders  rather  than  in  driving  architecture  development.  “As  the 
ATAM/QAW  was  exercised  late  in  [the]  development  process,  it  verified  and  validated  the 
quality  of  the  architecture  to  a  larger  set  of  representatives  within  the  user  space.  Secondari¬ 
ly,  the  ATAM/QAW  identified  a  consistent  set  of  risk  themes  for  tracking  by  the  govermnent 
and  suppliers  teams.” 

In  the  one  case  where  the  ATAM/QAW  was  reported  to  be  too  early  in  the  life  cycle,  the  benefit 
was  reported  to  be  equal  to  its  cost  and  the  engagement  resulted  in  a  positive  impact  on  the  “archi¬ 
tecture  construction.”  When  done  too  late,  the  ATAM/QAW  still  delivered  value  in  validating 
decisions  that  have  been  made,  although  conducting  AT  AMs  earlier  in  the  process  would  more 
readily  accommodate  “architectural  changes.” 

3.3  Need  and  Motivation 

The  reasons  that  programs  elected  to  pursue  the  ATAM/QAW  varied.  Generally,  reasons  for  con¬ 
ducting  the  methods  are  linked  with  the  acquisition  and  development  processes,  especially  for 
dealing  with  change,  managing  risk,  and  establishing  priorities.  Some  of  the  reasons  given  for 
pursuing  the  methods  included  the  following: 

•  “To  support  PMO  decision  making” 

•  “It  was  determined  that  sufficient  stakeholders  had  changed  and  there  was  a  need  to  re-assess 
the  Quality  Attribute  Tree  and  related  scenarios.” 

•  “The  ATAM  scenario  development  activity  helped  focus  available  funds  on  the  most  impor¬ 
tant  items.” 

•  “The  peer  review  process  [meaning  the  ATAM]  helped  to  identify  and  prioritize  the  most 
critical  aspects  to  transition  customers.” 

Honest  objectives  alone  are  not  sufficient  for  maximal  impact  of  the  methods,  however.  Having  a 
genuine  need  to  conduct  an  ATAM/QAW  versus  taking  advantage  of  an  Army-sponsored  (and 
funded)  opportunity  to  conduct  one  can  significantly  affect  the  perceived  benefit  and  impact  of 
conducting  the  ATAM/QAW.  Using  the  methods  without  first  determining  that  they  are  suitable 
for  the  circumstances  offers  only  a  hit-or-miss  opportunity  for  receiving  value  from  them.  Thus, 
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need  corresponds  to  whether  the  ATAM/QAW  was  appropriate  for  the  situation  or  whether  some 
other  analysis  technique  would  have  been  more  desirable.5 

There  was  an  appropriate  need  in  most  of  the  programs.  Factors  that  stressed  the  application  of  the 
methods  included  the  need  to  manage  systems  of  systems,  multiple  platforms,  or  research  and 
development  projects. 

•  “[SoS]  nonfunctional  requirements  are  much  broader  than  can  be  handled  in  a  reasonable  set 
of  QA  scenarios.” 

•  One  program  “created  a  specific  Arch  Evaluation  process,  leveraging  the  AT  AM  but  with 
significant  modifications”  in  order  to  better  meet  its  particular  needs. 

•  “Architecture  was  established  on  multiple  platforms  .  .  .  Since  commonality  across  platforms 
is  important  to  cost  containment  agreement  of  ALL  platforms  would  be  required  to  make  vi¬ 
able  changes  to  the  architecture.  A  users  group  is  and  was  in  existence  at  which  many  of  the 
similar  issues  are  raised  and  discussed  for  possible  implementation.” 

•  “As  this  was  a  developmental  system  under  examination,  the  project  was  not  under  any  par¬ 
ticular  pressure  to  exit  with  ‘fieldable’  software.” 

•  “I  think  the  adoption  of  the  practices  is  a  beneficial  undertaking;  however,  this  AT  AM  was 
applied  to  a  (probably  one-time)  research  and  development  project  that  was  close  to  its  dem¬ 
onstration  phase.  Therefore,  the  practices  were  not  incorporated.  If  I  was  starting  over,  I 
would  consider  the  ATAM  process  very  favorably.” 

Using  ATAM/QAW  at  the  systems  level  provided  some  benefits  but  pointed  to  other  techniques 
that  are  needed  (e.g.,  methods  to  elicit  mission  threads  and  methods  to  evaluate  systems  and  sys¬ 
tems  of  systems).  Research  and  development  projects  reported  value  during  the  execution  of  the 
methods  but  transition  efforts  were  not  applicable  in  all  cases. 

No  less  important  than  an  established  need  is  a  level  of  enthusiasm  for  the  evaluations.  The  pro¬ 
gram  office  needs  to  be  appropriately  motivated  to  arrange  for  and  conduct  the  ATAM/QAW  or  it 
will  likely  be  problematic.  If  the  program  office  only  tacitly  or  halfheartedly  supports  the  en¬ 
gagement,  it  will  be  difficult  to  motivate  key  stakeholders  to  actively  participate  and  make  an 
earnest  effort  to  contribute.  Moreover,  the  program  office  needs  someone  to  champion  the  effort 
and  persevere  in  making  the  contractual  arrangements  and  bringing  the  key  stakeholders  together 
at  an  opportune  time.  Training  members  of  the  program  office  or  its  representatives  as  ATAM 
Evaluators  so  that  they  can  be  part  of  the  ATAM  evaluation  team  would  enable  them  to  leam 
first-hand  the  value  of  such  engagements  and  provide  motivation  to  transition  the  practices. 

Where  motivation  among  all  stakeholders  was  highest  (in  58%  of  the  programs),  comments  such 
as  the  following  were  not  uncommon. 

•  “An  important  factor  leading  to  the  success  of  these  activities  was  a  motivated  and  architec¬ 
turally  educated  management  and  supplier  team.  Because  the  team  fully  accepted  the  impor- 


The  SEI  has  a  variety  of  architecture-centric  techniques.  Among  others,  the  SEI  Mission  Thread  Workshop 
(MTW)  elicits  quality  attribute  concerns  relative  to  mission  threads  from  a  system  of  systems  perspective.  The 
SEI  Architecture  Improvement  Workshop  (AIW)  is  a  useful  follow-on  to  an  ATAM  evaluation  that  begins  with  the 
identified  architectural  risks  and  explores  ways  of  mitigating  them  in  a  rational,  prioritized  fashion.  The  SEI  Cost 
Benefit  Analysis  Method  (CBAM)  is  a  way  of  analyzing  the  costs,  benefits,  and  schedule  implications  of  archi¬ 
tectural  decisions,  and  can  be  used  by  itself  or  in  conjunction  with  an  ATAM  evaluation. 
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tance  of  identifying  nonfunctional  or  quality  based  architectural  driving  requirements  it  al¬ 
lowed  for  a  more  thorough  assessment  of  the  architecture  for  trade  space  among  the  re¬ 
quirements  and  ultimately  for  its  support  to  the  identified  set  of  quality  attributes.” 

•  “The  ATAM/QAW  process  and  report  provided  both  the  government  program  office  and  the 
contracted  supplier  the  confidence  and  stakeholder  support  to  address  follow-on  user  re¬ 
quirements  that  far  exceeded  the  original  capabilities  as  implemented  within  the  architec¬ 
ture.” 

Motivation  (or  lack  of  motivation)  was  related  to  education  in  architecture  principles  (on  the  part 
of  PMO  staff,  contractors,  and  stakeholders),  confidence  that  the  methods  would  yield  some  posi¬ 
tive  result,  sufficient  flexibility  in  cost  and  scheduling  to  make  using  the  results  possible,  and  the 
existence  of  formal  or  informal  agreements  between  the  PMO  and  contractors  to  perform  follow 
up  activities.  The  following  comments  are  illustrative. 

•  “It  appeared  to  the  stakeholders  that  these  activities  did  not  warrant  action  since  neither  the 
time  nor  funds  were  readily  available  to  support  them.  In  addition,  it  was  very  questionable 
as  to  any  impact  those  activities  might  have.”  Stakeholders  of  this  program  doubted  there 
was  sufficient  flexibility  in  the  program  to  follow  through  on  the  evaluation  outcomes  and 
dubious  about  the  potential  benefits. 

•  “No  mandate  to  [use  quality  attribute  scenarios]  with  more  robust  requirements  management 
and  clarifying  nonfunctional  requirements.”  On  more  than  one  program,  absence  of  agree¬ 
ment  between  the  PMO  and  contractor  prevented  full  commitment  to  make  use  of  evaluation 
results. 

•  “The  architecture  was  never  really  taken  seriously  at  the  contractor  facility.”  Here  again, 
absence  of  agreement  led  to  failure  to  follow  through. 

•  “It  is  very  difficult  to  change  managements’  perceptions  and  the  current  way  of  doing  busi¬ 
ness  in  the  government.” 

•  “There  are  two  hindrances  to  doing  more.  First  is  limited  understanding  of  the  value  outside 
the  software  community  of  ‘software  things’.  .  .  .  The  larger  hindrance  is  that  PMs  are 
funded  to  meet  system  requirements  (from  requirements  documents)  in  accordance  with 
process  requirements  (from  DoD  5000).  PMs  are  not  funded  to  do  things  outside  those  re¬ 
quirements,  even  if  they  are  great  ideas.” 

3.4  Follow  Through 

The  ATAM  identifies  risks  that  need  to  be  integrated  into  both  the  program  office’s  and  the  sys¬ 
tem  developer’s  standard  risk  management  systems,  so  that  they  are  appropriately  tracked  and 
mitigated.  Moreover,  it  is  desirable  that  these  architecture  practices  be  considered  for  longer  term 
adoption  as  part  of  the  program  office’s  standard  acquisition  practices.  Longer-term  adoption  pro¬ 
vides  additional  benefits  by  leveraging  the  extensive  training  Army  personnel  have  received  in 
architecture  practices  and  enabling  lessons  learned  and  experiences  to  be  shared  across  Army  pro¬ 
grams.  In  turn,  these  additional  benefits  provide  an  effective  basis  for  initiating  a  program  to 
evolve  and  improve  acquisition  practices  for  Army  programs. 

There  are  many  reasons  why  there  may  be  a  lack  of  follow  through,  such  as 

•  a  political  environment  that  prevents  formally  tracking  new  risks 
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•  inadequate  expertise  in  the  program  office  to  assess  contractor  response  to  engagement  re¬ 
sults 

•  inadequate  expertise  at  the  contractor  to  deal  with  findings  of  the  engagement 

•  a  funding  profile  that  does  not  support  adequate  upfront  engineering  effort 

Follow  through  was  high  in  42%  of  the  programs.  Successful  follow  through  was  accomplished 
because  processes  were  already  in  place  (or  put  in  place)  to  use  the  results,  several  having  to  do 
with  risk  management. 

•  “The  supplier  treated  each  risk  theme  as  a  trouble  report  and  attempted  to  resolve  each  in  a 
formal  process  with  the  Government  Architecture  team.” 

•  “The  adopted  practices  were  incorporated  into  the  program  SDP  based  on  recognition  of  the 
importance  of  software  architecture  practices  to  program  success.” 

•  “The  resulting  list  of  QAs  was  flowed  down  to  lower  tier  suppliers  and  required  for  their 
architectures  to  address.” 

•  “Many  of  these  activities  were  already  planned  but  the  ATAM  results  were  incorporated  into 
the  processes.  Risk  management  is  a  required  activity  and  once  risks  are  identified  they  must 
be  tracked  and  mitigation  strategies  developed.  Once  something  is  in  the  database,  it  must 
have  a  mitigation  strategy  and  be  tracked.” 

•  “For  software,  quality  attribute  requirements  and  an  AT  AM-based  plan  for  acceptable  satis¬ 
faction  of  quality  attributes  were  specified  and  tracked  in  non-traditional  but  binding  RFP 
documentation.” 

.  “In  most  cases  suppliers  and  government  teams  identified  products  and  processes  leveraged 
quality  attribute  identification,  prioritization,  modeling,  implementation,  and  test  that  reflect¬ 
ed  the  intent  of  the  ATAM  and  QAW  processes.” 

•  “[The  program]  has  a  risk  management  process  including  risk  identification  and  risk  mitiga¬ 
tion  planning.  Addressing  architecture  related  risks  are  part  of  this  process.” 

In  two  cases,  follow  through  was  seen  as  a  necessary  step  not  only  to  ensure  appropriate  architec¬ 
tural  decisions  but  also  to  further  understand  system  requirements  in  general. 

•  “Follow-on  refinement  of  QAW  scenarios  and  generation  of  new  scenarios  were  necessary 
to:  (1)  include  the  perspective  (scenarios  and  prioritization)  of  stakeholders  not  present  at  the 
QAW,  (2)  validate/substantiate  QAW  scenarios  with  system  architecture  and  system  re¬ 
quirements,  and  (3)  to  incorporate  changing  program  business  goals  and  requirements  prior 
to  the  ATAM  and  RFP  release.” 

•  “Additional  scenarios  were  felt  to  be  important ....  Software  architecture  documentation 
was  revised  to  improve  the  clarity.  The  ATAM  is  now  commonly  cited  in  reviews  as  a  vali¬ 
dation  of  proper  understanding  and  refinement  of  stated  requirements.” 

As  noted  previously,  contractual  impediments  were  a  major  obstacle  to  follow  through  on  the 
evaluation  results. 
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3.5  Other  Factors 


In  addition  to  the  predefined  contextual  factors  noted  above,  survey  responses  indicated  other  fac¬ 
tors  that  affected  the  impact  of  the  ATAM/QAW,  mostly  having  to  do  with  people  and  organiza¬ 
tional  issues. 

The  training  and  expertise  of  the  evaluation  team  (which,  in  most  cases,  consisted  of  SEI  facilita¬ 
tors  and  a  mix  of  SEI  and  SEI-trained  Army  evaluators)  was  cited  often.  Insufficient  levels  of 
trained  staff  were  a  clear  barrier  to  continued  implementation  of  the  methods. 

•  “Excellent  training  and  access  to  very  qualified  AT  AM  facilitators.” 

•  “[The  program  gained]  valuable  experience  .  .  .  from  government  team  (participation  on 
evaluation  team).” 

•  “The  program  office  does  not  have  sufficient  numbers  of  govermnent  employees  as  Core  or 
Matrix  support  to  fully  and  properly  support  QAW  and  AT  AM  efforts  for  the  life  of  the  pro¬ 
gram.” 

•  “In  the  past  SEI  architecture-level  training  courses  have  been  identified  but  due  to  schedul¬ 
ing  constraints  could  not  be  attended  by  members  of  the  team.” 

Although  there  is  certainly  interest  in  the  methods  among  the  Army’s  software  professionals, 
long-term  adoption  is  limited  by  a  (perceived)  lack  of  need. 

•  “[The  program]  does  not  have  any  new  architecture -based  contracts,  so  there  hasn’t  been  a 
perceived  need  to  establish  an  ATAM/QAW  for  new  projects.” 

•  “In  addition  to  other  cited  hindrances,  the  project  is  the  only  software-intensive  system  with¬ 
in  PM.  Any  additional  contracts  are  only  follow-on  extensions.  There  are  no  new  programs 
coming  online  at  this  time.  Additional  people  have  been,  and  continue  to  be,  trained  as  soft¬ 
ware  architects  and  evaluators  but  this  is  done  through  their  parent  organization  as  profes¬ 
sional  development  rather  than  through  the  PM  to  which  they  are  matrixed  as  support.  This 
training  is  encouraged  from  supervisors  in  the  PM  organization  when  the  schedule  permits.” 

Note  that  many  programs  obtain  software  engineering  support  from  one  of  the  Army’s  software 
engineering  centers  (SECs).  The  existence  of  these  centers  suggests  that  the  lack  of  need  is  a  per¬ 
ceived  state  of  affairs  and  that  awareness  of  non-software  program  staffs  needs  to  be  raised. 
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4  Conclusions 


This  special  report  describes  the  results  of  a  study  of  the  impact  of  software  architecture  practices 
conducted  with  Army  programs.  Twelve  programs  that  employed  the  ATAM  and/or  QAW  re¬ 
sponded  to  a  questionnaire  that  addressed  the  impact  of  conducting  the  method,  follow-on  activi¬ 
ties,  adoption  of  the  method  as  part  of  program  practices,  and  the  overall  value  of  the  engagement. 
The  consensus  of  the  respondents  was  that  they  received  value  from  the  QAWs  and/or  ATAM 
evaluations  conducted  for  their  programs. 

Value  can  be  understood  in  terms  of  programmatic,  product,  and  practice  improvements.  Given 
the  appropriate  conditions,  the  ATAM  and  QAW 

•  are  cost-effective  techniques  for  evaluating  software  architectures  with  respect  to  the  archi¬ 
tecturally  significant  requirements 

•  provide  a  favorable  impact  on  the  understanding  of  three  key  development  artifacts:  the 
quality  attribute  requirements  of  the  system,  architectural  design  decisions,  and  risks  driving 
product  development 

•  foster  communication  among  stakeholders  and  provide  an  infonned  basis  for  the  program 
office,  suppliers,  and  stakeholders  to  communicate  their  needs  and  understand  how  they  are 
met 

Analysis  revealed  that  the  measure  of  value  is  influenced  by  the  context  in  which  the  practices  are 
used.  Programs  in  more  favorable  contexts  expressed  higher  perceptions  of  ATAM/QAW  value 
than  programs  less  favorably  situated.  It  is  interesting  to  note  that  all  programs  were  able  to  find 
some  value  in  their  respective  experiences.  In  reviewing  the  lessons  learned,  certain  patterns  of 
circumstances  emerge  as  leading  to  higher  impact  outcomes.  These  patterns  can  be  understood  in 
terms  of  the  system  acquisition  life  cycle  that  situates  the  methods  so  that  the  appropriate  entry 
conditions  are  met,  providing  favorable  circumstances  for  their  application. 

The  greatest  impediment  to  reaping  the  maximum  benefit  from  the  ATAM  architecture  evalua¬ 
tions  conducted  on  the  Army  systems  indentified  in  this  report  is  that  the  evaluations  were  primar¬ 
ily  done  in  a  reactive  mode  (i.e.,  the  evaluations  were  conducted  opportunistically  and  performed 
under  an  existing  contract  as  opposed  to  being  pre-planned  and  incorporated  into  the  RFP/contract 
from  the  beginning  of  the  acquisition).  The  findings  provide  lessons  learned  for  improving  the 
context  through  proactive  planning,  improved  execution,  and  organizational  and  process  support 
for  follow  through. 

The  U.S.  Army’s  Warfighter  Information  Network-Tactical  (WIN-T)  was  one  of  the  programs 
surveyed  and  provides  a  good  example  of  what  is  possible  given  mostly  desirable  ratings  for  the 
contextual  factors  that  influence  the  impact  of  the  architectural  practices.  A  detailed  case  study 
has  been  written  that  presents  the  WIN-T  program  context,  describes  the  application  of  the 
ATAM  to  the  WIN-T  system,  presents  important  results,  and  summarizes  the  benefits  the  program 
received  [Clements  2005],  Planning  was  reactive,  requiring  accommodation  on  the  part  of  the 
program  office  and  contractors  to  modify  the  task  execution  plan,  but  the  other  contextual  factors 
were  favorable.  The  case  study  provides  an  example  of  how  a  government  organization  can  incor¬ 
porate  technologies  such  as  ATAM  to  solve  real  problems  and  improve  its  mission  effectiveness 
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in  a  government-owned,  contractor-operated  environment.  A  development  that  attests  to  the  bene¬ 
fits  the  WIN-T  program  perceived  they  received  is  that  the  WIN-T  program  conducted  another 
ATAM  later  in  the  life  cycle  when  the  program  was  re-defined  and  volunteered  to  pilot  the  new 
system  and  software  ATAM  that  concurrently  evaluates  both  the  system  and  software  architec¬ 
ture. 

To  achieve  maximum  effectiveness  in  reducing  software  acquisition  risk,  an  acquisition  organiza¬ 
tion  should  be  proactive  in  applying  architecture-centric  practices.  In  a  proactive  approach,  archi¬ 
tecture-centric  practices,  such  as  a  QAW  and  the  ATAM  software  architecture  evaluation,  are 
preplanned  and  integrated  up  front  into  the  RFP/contract. 

Being  proactive  allows  the  government  to  conduct  a  QAW  with  government  stakeholders  during 
the  RFP  planning  phase.  The  result  of  a  QAW  allows  the  quality  attribute  requirements  to  be  in¬ 
cluded  in  the  RFP,  so  that  they  can  appropriately  drive  the  contractor’s  architectural  design  ap¬ 
proach.  Following  contract  award,  another  QAW  can  be  held  in  collaboration  with  contractor 
stakeholders.  This  allows  for  refinement,  clarification,  and  expansion  of  quality  attribute  require¬ 
ments  in  conjunction  with  contractor  stakeholders  such  as  the  chief  software  architect,  domain 
experts,  and  software  developers. 

An  effective  way  to  ensure  that  the  proposed  architectural  design  is  suitable  for  achieving  the  spe¬ 
cified  system  qualities  is  to  require  that  an  ATAM  architecture  evaluation  be  conducted  prior  to  a 
key  contractual  event  such  as  a  traditional  DoD  PDR  or  CDR. 

The  funding  provided  by  ASSIP  was  the  impetus  for  conducting  these  evaluations;  without  that 
funding,  it  is  unlikely  that  they  would  have  taken  place.  If  architecture  evaluation  is  to  become 
routine  practice  within  the  Army,  ASSIP  needs  to  continue  to  play  a  leadership  role  to  provide  the 
context  that  will  enable  Army  programs  to  proactively  apply  architecture-centric  acquisition  prac¬ 
tices. 

It  is  recommended  that  future  studies  be  conducted  to  measure  the  impact  of  architecture  evalua¬ 
tions  that  are  conducted  proactively.  Currently,  we  know  of  two  DoD  programs  that  proactively 
included  an  architecture  evaluation  in  RFP/contracts,  one  of  which  is  an  Army  program.  A  diffi¬ 
culty  highlighted  in  questionnaire  responses  is  the  belief  that  ATAM  evaluations  will  not  be  rou¬ 
tinely  performed  on  Army  programs,  especially  in  a  proactive  mode,  unless  Army  policy  man¬ 
dates  it  for  programs  such  as  Acquisition  Category  (ACAT)  1  programs. 
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Appendix  A  Acronym  List 


ABCS 

Army  Battle  Command  System 

ACAT 

Acquisition  Category 

ACS 

Aerial  Common  Sensor 

AIW 

Architecture  Improvement  Workshop 

APW 

Acquisition  Planning  Workshop 

ASS  IP 

Army  Strategic  Software  Improvement  Program 

ATAM 

Architecture  Tradeoff  Analysis  Method 

CAAS 

Common  Avionics  Architecture  System 

CBAM 

Cost  Benefit  Analysis  Method 

CDR 

critical  design  review 

CECOM 

Communications  and  Electronics  Command 

CMMI 

Capability  Maturity  Model  Integration 

CONOPS 

concept  of  operations 

COTS 

commercial  off-the-shelf 

CPoF 

Command  Post  of  the  Future 

DCGS-A 

Distributed  Common  Ground  Station  -  Army 

DoD 

Department  of  Defense 

DoDAF 

Department  of  Defense  Architecture  Framework 

FBCB2 

Force  XXI  Battle  Command,  Brigade-and-Below 

FCS 

Future  Combat  System 

GOTS 

government  off-the-shelf 

IFC 

Integrated  Fired  Control 

IPT 

integrated  product  team 

JTCW 

Joint  Tactical  Common  Operational  Picture  Workstation 

KPP 

key  performance  parameter 

MCAP 

Manned/Unmanned  Common  Architecture  Program 

MTW 

Mission  Thread  Workshop 

OneSAF 

One  Semi-Automated  Forces 

PD 

project  director 

PDR 

preliminary  design  review 

PEO 

Program  Executive  Office 

PM 

program  manager 

PMO 

program  management  office 

QAW 

Quality  Attribute  Workshop 

ROI 

Return  on  Investment 

RFP 

request  for  proposal 

SDP 

System  Design  Plan 

SEC 

software  engineering  center 

SED 

Software  Engineering  Directorate 

SEI 

Software  Engineering  Institute 

SoS 

system  of  systems 

SW 

software 

TCM 

TRADOC  Capabilities  Manager 

TRADOC 

Training  and  Doctrine  Command 

TSM 

TRADOC  Systems  Manager 

Win-T 

Warfighter  Information  Network-Tactical 
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Appendix  B  About  the  SEI  AT AM  and  QAW 


The  two  architecture  practices  that  were  applied  in  the  engagements  with  the  selected  Army  pro¬ 
grams  were  the  ATAM  and  QAW. 

The  Architecture  Tradeoff  Analysis  Method  (ATAM) 

The  purpose  of  the  ATAM  is  to  assess  the  consequences  of  architectural  decision  alternatives  in 
light  of  quality  attribute  requirements  [Kazman  2000].  The  major  goals  of  the  ATAM  are  to 

•  elicit  and  refine  a  precise  statement  of  the  architecture’s  driving  quality  attribute  require¬ 
ments 

•  elicit  and  refine  a  precise  statement  of  the  architectural  design  decisions 

•  evaluate  the  architectural  design  decisions  to  determine  if  they  address  the  quality  attribute 
requirements  satisfactorily 

The  ATAM  is  predicated  on  the  fact  that  an  architecture  is  suitable  (or  not  suitable)  only  in  the 
context  of  specific  quality  attributes  that  it  must  impart  to  the  system.  The  ATAM  uses  stakehold¬ 
er  perspectives  to  produce  a  collection  of  scenarios  that  define  the  qualities  of  interest  for  the  par¬ 
ticular  system  under  consideration.  Scenarios  give  specific  instances  of  usage,  performance  re¬ 
quirements,  growth  requirements,  various  types  of  failures,  various  possible  threats,  and  various 
likely  modifications.  Once  the  important  quality  attributes  are  identified  in  detail,  the  architectural 
decisions  relevant  to  each  one  can  be  illuminated  and  analyzed  with  respect  to  their  appropriate¬ 
ness. 

The  steps  of  the  ATAM  are  carried  out  in  two  main  phases.  In  the  first  phase,  the  evaluation  team 
interacts  with  the  system’s  primary  decision  makers:  the  architect(s),  manager(s),  and  perhaps  a 
marketing  or  customer  representative.  During  the  second  phase,  a  larger  group  of  stakeholders  is 
assembled,  including  developers,  testers,  maintainers,  administrators,  and  users.  The  two-phase 
approach  insures  that  the  analysis  is  based  on  a  broad  and  appropriate  range  of  perspectives.6 

Phase  1 

1 .  Present  the  ATAM.  The  evaluators  explain  the  method  so  that  those  who  will  be  involved  in 
the  evaluation  have  an  understanding  of  the  ATAM  process. 

2.  Present  business  drivers.  The  appropriate  system  representatives  present  an  overview  of  the 
system,  its  requirements,  business  goals,  context,  and  the  architectural  quality  drivers. 

3.  Present  architecture.  The  system  or  software  architect  (or  another  lead  technical  person) 
presents  the  architecture. 

4.  Catalog  architectural  approaches.  The  system  or  software  architect  presents  general  architec¬ 
tural  approaches  to  achieve  specific  qualities.  The  evaluation  team  captures  a  list  and  adds  to 
it  any  approaches  they  saw  during  Step  3  or  learned  during  their  pre-exercise  review  of  the 
architecture  documentation.  For  example,  “a  cyclic  executive  is  used  to  ensure  real-time  per- 

6  These  two  phases  are  sandwiched  by  two  less  intensive  phases.  Phase  0  is  a  preparation  phase  in  which  the 
evaluation  activities  are  planned  and  set  up.  Phase  3  is  a  follow-up  phase  in  which  the  final  report  is  produced 
and  opportunities  for  improving  the  process  are  considered. 
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formance.”  Known  architectural  approaches  have  known  quality  attribute  properties  that  will 
help  in  carrying  out  the  analysis  steps. 

5.  Generate  a  quality  attribute  utility  tree.  Participants  build  a  utility  tree,  which  is  a  prioritized 
set  of  detailed  statements  about  what  quality  attributes  are  most  important  for  the  architecture 
to  achieve  (such  as  performance,  modifiability,  reliability,  or  security)  and  specific  scenarios 
that  express  these  attributes. 

6.  Analyze  architectural  approaches.  The  evaluators  and  the  architect(s)  map  the  utility  tree  sce¬ 
narios  to  the  architecture  to  see  how  it  responds  to  each  one. 

Phase  2 

Phase  2  begins  with  an  encore  of  the  Step  1  AT  AM  presentation  and  a  recap  of  the  results  of 
Steps  2  through  6  for  the  larger  group  of  stakeholders.  Then  these  steps  are  followed: 

1 .  Brainstorm  and  prioritize  scenarios.  The  stakeholders  brainstorm  additional  scenarios  that 
express  specific  quality  concerns.  After  brainstorming,  the  group  chooses  the  most  important 
ones  using  a  facilitated  voting  process. 

2.  Analyze  architectural  approaches.  As  in  Step  6,  the  evaluators  and  the  architect(s)  map  the 
high-priority  brainstormed  scenarios  to  the  architecture. 

3.  Present  results.  A  presentation  is  produced  that  captures  the  results  of  the  process  and  sum¬ 
marizes  the  key  findings  that  are  indicative  of  what  will  be  in  the  final  report  (a  product  of 
Phase  3). 

Scenario  analysis  produces  the  following  results: 

•  a  collection  of  sensitivity  and  tradeoff  points.  A  sensitivity  point  is  an  architectural  decision 
that  affects  the  achievement  of  a  particular  quality.  A  tradeoff  point  is  an  architectural  deci¬ 
sion  that  affects  more  than  one  quality  attribute  (possibly  in  opposite  ways). 

•  a  collection  of  risks  and  non-risks.  A  risk  is  an  architectural  decision  that  is  problematic  in 
light  of  the  quality  attributes  that  it  affects.  A  non-risk  is  an  architectural  decision  that  is  ap¬ 
propriate  in  the  context  of  the  quality  attributes  that  it  affects. 

•  a  list  of  current  issues  or  decisions  not  yet  made.  Often  during  an  evaluation,  issues  not  di¬ 
rectly  related  to  the  architecture  arise.  They  may  have  to  do  with  an  organization’s  processes, 
personnel,  or  other  special  circumstances.  The  AT  AM  process  records  these  issues,  so  they 
can  be  addressed  by  other  means.  The  list  of  decisions  not  yet  made  arises  from  the  stage  of 
the  system  life  cycle  during  which  the  evaluation  takes  place.  An  architecture  represents  a 
collection  of  decisions.  Not  all  relevant  decisions  may  have  been  made  at  the  time  of  the 
evaluation,  even  when  designing  the  architecture.  Some  of  these  decisions  are  known  to  the 
development  team  as  having  not  been  made  and  are  on  a  list  for  further  consideration.  Others 
are  news  to  the  development  team  and  stakeholders. 

Results  of  the  overall  exercise  also  include  the  summary  of  the  business  drivers,  the  architecture, 
the  utility  tree,  and  the  analysis  of  each  chosen  scenario.  All  of  these  results  are  recorded  visibly 
so  all  stakeholders  can  verify  that  they  have  been  identified  correctly. 

The  number  of  scenarios  analyzed  during  the  evaluation  is  controlled  by  the  amount  of  time  al¬ 
lowed  for  the  evaluation,  but  the  process  insures  that  the  most  important  ones  are  addressed. 
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After  the  evaluation,  the  evaluators  write  a  report  documenting  the  evaluation  and  recording  the 
information  discovered.  This  report  also  documents  the  framework  for  ongoing  analysis  discov¬ 
ered  by  the  evaluators.  Clements,  Kazman,  and  Klein  provide  detailed  descriptions  of  the  AT  AM 
process  [Kazman  2000,  Clements  2002]. 

The  Quality  Attribute  Workshop  (QAW) 

The  QAW  is  a  facilitated  method  that  engages  system  stakeholders  early  in  the  life  cycle  to  dis¬ 
cover  the  driving  quality  attributes  of  a  software-intensive  system.  It  provides  a  means  to  gener¬ 
ate,  prioritize,  and  refine  quality  attribute  scenarios  before  the  software  architecture  is  completed. 
The  QAW  is  focused  on  system-level  concerns  and  specifically  the  role  that  software  will  play  in 
the  system. 

The  QAW  involves  the  following  steps: 

1 .  QAW  Presentation  and  Introductions.  The  facilitator  explains  the  method  so  that  those  who 
will  be  involved  in  the  workshop  have  an  understanding  of  the  QAW  process. 

2.  Business/Mission  Presentation.  The  appropriate  system  representatives  present  an  overview 
of  the  system,  its  requirements,  business  goals,  context,  and  the  architectural  quality  drivers. 

3.  Architectural  Plan  Presentation.  The  system  or  software  architect  (or  another  lead  technical 
person)  presents  the  architectural  plan. 

4.  Identification  of  Architectural  Drivers.  The  facilitation  team  captures  information  regarding 
architectural  drivers  that  are  key  to  realizing  quality  attribute  goals  in  the  system.  These  driv¬ 
ers  often  include  high-level  requirements,  business/mission  concerns,  goals  and  objectives, 
and  various  quality  attributes. 

5.  Scenario  Brainstorming.  The  stakeholders  brainstorm  scenarios  that  express  specific  quality 
concerns. 

6.  Scenario  Consolidation.  The  stakeholders  consolidate  similar  scenarios  before  they  are  priori¬ 
tized. 

7.  Scenario  Prioritization.  The  stakeholder  chooses  the  most  important  scenarios  using  a  facili¬ 
tated  voting  process. 

8.  Scenario  Refinement.  The  stakeholders  further  elaborate  the  high-priority  brainstormed  sce¬ 
narios. 

Scenario  refinement  produces  the  following  results: 

•  Business  and  mission  goals  affected  by  the  scenario. 

•  Quality  attributes  associated  with  the  scenario. 

•  A  concrete  description  of  the  scenario  in  terms  of:  (1)  the  stimulus  that  affects  the  system; 

(2)  the  response  that  results  from  the  stimulus;  (3)  the  entity  that  generated  the  stimulus;  (4) 
the  environment  under  which  the  stimulus  occurred;  (5)  the  artifact  that  was  stimulated;  and 
(6)  the  measure  by  which  the  system’s  response  will  be  evaluated. 

•  Questions  and  issues  raised  by  the  stakeholders  regarding  the  scenario.  Such  questions  and 
issues  concentrate  on  the  quality  attribute  aspects  of  the  scenario  and  any  concerns  that  the 
stakeholders  might  have  in  achieving  the  response  called  for  in  the  scenario. 
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Results  of  the  overall  exercise  also  include  the  summary  of  the  business  drivers,  the  architectural 
plan,  the  collection  of  brainstormed  scenarios,  and  the  refinement  of  each  chosen  scenario.  All  of 
these  results  are  recorded  visibly  so  all  stakeholders  can  verify  that  they  have  been  identified  cor¬ 
rectly. 

The  number  of  scenarios  refined  during  the  workshop  is  controlled  by  the  amount  of  time  allowed 
for  the  refinement,  but  the  process  insures  that  the  most  important  ones  are  addressed. 

After  the  workshop,  the  facilitators  write  a  report  documenting  the  workshop  and  recording  the 
information  discovered.  Barbacci  and  other  provide  a  description  of  the  QAW  process  [Barbacci 
2003], 
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Appendix  C  ATAM/QAW  Impact  Questionnaire 


June  23, 2008 


RE:  [system]  [QAW/ATAM]  held  [date] 


The  Office  of  the  Assistant  Secretary  of  the  Army  (Acquisition,  Logistics,  and  Technology)  has 
asked  us  to  interview  personnel  from  programs  that  have  had  Architecture  Tradeoff  Analysis  Me¬ 
thod®  (AT AM®)  or  Quality  Attribute  Workshop  (QAW)  engagements  with  the  Software  Engi¬ 
neering  Institute  (SEI).  Our  goal  in  contacting  you  is  to  gauge  the  impact  that  the  engagement  had 
on  the  quality  of  the  system  and  the  practices  of  your  organization.  In  order  to  understand  impact, 
we  are  collecting  follow-on  data  from  all  of  the  participating  programs  via  a  short  questionnaire 
and  a  subsequent  interview. 

We  are  asking  you  to  complete  the  enclosed  questionnaire  and  return  it  within  two  weeks.  We 
estimate  it  will  take  no  more  than  an  hour.  For  your  convenience,  you  may  either  edit  the  Micro¬ 
soft  Word  file  directly  or  print  out  a  copy  and  fill  in  the  responses.  If  you  edit  the  document  on¬ 
line,  you  can  highlight  an  empty  check  box,  and  type  the  letter  “x”  to  check  the  box.  Please  take 
time  to  respond  to  the  open-ended  questions.  These  provide  an  opportunity  for  you  to  explain  the 
rationale  for  your  checked  responses. 

Please  be  candid  in  your  responses  and  complete  this  questionnaire  as  best  you  can.  If  you  are 
uncertain  about  an  answer,  or  if  you  are  reporting  the  view  of  a  colleague,  please  indicate  this  as 
part  of  your  response.  We  are  looking  for  your  best  reasoned  estimates  and  responses. 

We  will  aggregate  and  analyze  the  data  from  all  respondents.  The  results  will  be  described  in  a 
report  that  will  be  sent  to  our  sponsor,  Mr.  Robert  Schwenk,  Senior  Software  Acquisition  Manag¬ 
er,  ASA(ALT).  The  report  will  identify  the  collection  of  participating  programs  responding  to  the 
survey  but  your  specific  data  will  not  be  identified  with  your  specific  program.  Our  overall  goal  is 
to  assess  the  value  and  impact  of  architecture-centric  acquisition  practices  to  the  Army. 

Thanks  for  your  help.  Your  cooperation  is  important  and  we  value  your  feedback. 

John  Bergey,  Stephen  Blanchette,  Mark  Klein,  Robert  Nord 
Software  Engineering  Institute 
Enel:  questionnaire 
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This  questionnaire  has  four  sections  that  address  (1)  conducting  the  ATAM/QAW,  (2)  fol- 
low-on  ATAM/QAW  activities,  (3)  adoption  of  ATAM/QAW  as  part  of  program  practices, 
and  (4)  overall  value  of  the  engagement.  The  questions  in  these  sections  address  the  impact  of 
the  engagement  on  the  quality  of  the  system  and  the  practices  of  the  involved  program  office, 
stakeholders,  and  suppliers.7 

I.  Conducting  the  ATAM/QAW 

The  ATAM/QAW  produces  and  uses  quality  attribute  (nonfunctional)  requirements,  architec¬ 
ture  documentation,  and  architecture  risks. 


Please  indicate  the  extent  to  which  you  believe  that  conduct¬ 
ing  the  ATAM/QAW 

a.  Clarified  quality  attribute  requirements . 

b.  Discovered  new  quality  attribute  requirements . 

c.  Exposed  architecturally  significant  (high-priority  and  high- 

impact)  requirements . 

d.  Improved  the  understanding  of  the  architecture . 

e.  Described  new  views  or  architectural  approaches . 

f.  Exposed  key  design  decisions  that  provided  additional  insight 

into  the  architecture . 

g.  Clarified  understanding  of  existing  tradeoffs . 

h.  Discovered  new  tradeoffs . 

i.  Exposed  important  tradeoffs  that  impacted  achievement  of 

business  and  mission  goals . 

j .  Clarified  understanding  of  existing  risks . 

k.  Discovered  new  risks . 

l.  Exposed  high-priority  risks  that  impacted  achievement  of 

business  and  mission  goals . 

m.  Improved  the  architecture . 

n.  Fostered  communication  among  stakeholders . 

o.  Provided  an  informed  basis  for  the  program  office  and  the 

supplier  to  better  understand  and  control  the  software  devel¬ 
opment  cost  and  schedule . 

p.  Provided  an  informed  basis  for  the  program  office  to  specify 

quality  attribute  requirements,  understand  the  software  de¬ 
sign,  and  evaluate  systems  to  ensure  achievement  of  business 
and  mission  goals . 
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□ 
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Note  that  the  term  supplier  refers  to  the  organization  responsible  for  supplying  the  software,  which  could  be  a 
contractor,  subcontractor,  software  engineering  center,  or  other  software  development  organization. 
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q.  Provided  an  informed  basis  for  the  supplier  to  understand  □  □  □ 

quality  attribute  requirements,  use  them  to  make  and  evaluate 

architecture  decisions,  and  improve  the  architecture . 

r.  Provided  an  informed  basis  for  stakeholders  to  communicate  □  □  □ 

requirements  and  understand  how  they  were  represented  and 

met . 

s.  Other  (please  identify):  □  □  □ 


□ 

□ 

□ 


t.  Comment: 


II.  Follow-On  ATAM/QAW  Activities 

The  following  sections  provide  examples  of  activities  that  might  have  occurred  after  and  as  a 
consequence  of  the  ATAM/QAW  engagement. 


1.  Specification  and  Use  of  System  Quality  Attributes 


Please  check  whether  or  not  the  activity  was  conducted.  >  % 

a.  Additional  quality  attribute  scenarios  were  refined  that  were  identified  q  [_) 

during  the  QAW . 

b.  Additional  quality  attribute  scenarios  were  analyzed  that  were  identified  □  □ 

during  the  AT  AM . 

c.  Additional  quality  attribute  scenarios  were  identified  after  the  □  □ 

ATAM/QAW . 

d.  Quality  attribute  scenarios  were  incorporated  into  the  requirements  base-  □  □ 

line . 

e.  Quality  attribute  scenarios  were  put  into  a  requirements  tracking  system .  □  □ 

f.  Documentation  containing  the  quality  attribute  requirements  was  created  q  [_j 

or  improved . 

g.  Quality  attribute  scenarios  were  adopted  as  the  preferred  means  of  speci-  □  □ 

fying  the  system’s  nonfunctional  requirements . 

h.  Quality  attribute  scenarios  were  used  in  the  RFP,  and/or  in  negotiations  □  □ 

with  the  supplier . 

i.  Quality  attribute  scenarios  were  used  in  the  development  of  the  architec-  □  [_) 

ture . 

j.  Quality  attribute  scenarios  were  used  in  conducting  architecture  walk-  [_]  [_] 

throughs . 

k.  Other  (please  identify):  □  □ 


1.  Please  describe  why  you  conducted  the  activities  indicated  by  “yes.”  What  was  the  realized 
benefit?  What  factors  enabled  or  contributed  to  the  success  of  performing  these  activities? 


29  |  CMU/SEI-2009-SR-007 


m.  Please  describe  why  you  did  not  conduct  the  activities  indicated  by  “no.”  Even  though  not 
performed,  is  there  a  perceived  benefit  to  doing  so?  What  were  the  obstacles  that  hindered 
you  from  doing  so? 


2.  Documentation  and  Use  of  Software  Architecture 


Please  check  whether  or  not  the  activity  was  conducted.  >  z 

a.  The  improvements  to  the  architecture  were  incorporated  into  the  archi-  [_]  [_) 

tecture  documentation . 

b.  The  program  office  was  able  to  use  the  documented  architecture  more  □  □ 

effectively . 

c.  The  documented  architecture  was  used  to  evaluate  future/other  changes  □  □ 

to  the  architecture . 

d.  The  supplier  was  required  to  place  the  software  architecture  description  □  □ 

document  under  formal  configuration  management  control . 

e.  The  supplier  was  required  to  formally  deliver  the  software  architecture  □  □ 

description  document  to  the  program  office . 

f.  The  supplier  was  required  to  include  the  software  architecture  in  its  de-  □  [_) 

scriptions  of  bi-directional  traceability . 

g.  Other  (please  identify):  □  □ 


h.  Please  describe  why  you  conducted  the  activities  indicated  by  “yes.”  What  was  the  realized 
benefit?  What  factors  enabled  or  contributed  to  the  success  of  performing  these  activities? 


i.  Please  describe  why  you  did  not  conduct  the  activities  indicated  by  “no.”  Even  though  not 
performed,  is  there  a  perceived  benefit  to  doing  so?  What  were  the  obstacles  that  hindered 
you  from  doing  so? 
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3.  Identification  and  Management  of  Architecture  Risks 


Please  check  whether  or  not  the  activity  was  conducted.  >  z 

a.  Additional  risks  were  subsequently  identified  as  a  result  of  conducting  q  q 

additional  ATAM  analysis . 

b.  The  status  and  disposition  of  some  or  all  the  risks  that  were  identified  q  q 

were  tracked . 

c.  The  program  office  oversaw  the  mitigation  of  risks  and/or  risk  themes .  q  q 

d.  The  program  office  entered  risks  and  risk  themes  into  its  standard  risk  q  q 

management  system  and  processed  them  accordingly . 

e.  The  program  office  met  with  the  supplier  to  discuss  risks  and  risk  themes  q  q 

and  their  resolution . 

f.  Risks  and  risk  themes  were  discussed  during  PDR,  CDR  or  some  other  q  q 

program  technical  review . 

g.  The  risk  mitigation  results  were  documented  .  q  q 

h.  The  supplier  entered  risks  and  risk  themes  into  its  standard  risk  manage-  □  □ 

ment  system  and  processed  them  accordingly . 

i.  A  formal  risk  mitigation  plan  was  developed  to  describe  how  the  risks  q  q 

and  risk  themes  should  be  mitigated  . 

j.  The  supplier  was  required  to  conduct  an  architectural  walkthrough  to  q  q 

demonstrate  that  the  risks  were  appropriately  mitigated  . 

k.  The  supplier  was  required  to  identify  the  changes  to  the  architecture  de-  □  □ 

scription  document  and  the  architecture  documentation  was  updated  in 

accordance  with  the  risk  mitigation  results . 

l.  Identified  risks  were  successfully  mitigated  resulting  in  an  improved  ar-  q  q 

chitecture . 

m.  Other  (please  identify):  q  □ 


n.  Please  describe  why  you  conducted  the  activities  indicated  by  “yes.”  What  was  the  realized 
benefit?  What  factors  enabled  or  contributed  to  the  success  of  performing  these  activities? 


o.  Please  describe  why  you  did  not  conduct  the  activities  indicated  by  “no.”  Even  though  not 
performed,  is  there  a  perceived  benefit  to  doing  so?  What  were  the  obstacles  that  hindered 
you  from  doing  so? 
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III.  Adoption  of  ATAM/QAW  as  part  of  Program  Practices 

The  following  list  provides  examples  of  how  the  ATAM/QAW  engagement  might  have  af¬ 
fected  the  long-term  acquisition  practices  of  the  program  office,  program  office  stakeholders, 
system  stakeholders,  and/or  suppliers. 


Please  check  one  of  the  boxes  for  each  practice,  noting  whether 

you  have  adopted  the  practice,  plan  to  adopt  the  practice,  or  do 

not  plan  to  adopt  the  practice: 

a.  Quality  attribute  scenarios  will  be  (are  being)  adopted  as  the 

means  for  specifying  the  system’s  nonfunctional  requirements  . 

b.  A  Quality  Attribute  Workshop  (QAW)  will  be  (is  being)  con¬ 

ducted  with  program  stakeholders  to  specify  a  system’s  non¬ 
functional  requirements  . 

c.  A  software  architecture  description  document  will  be  (is)  a  re¬ 
quired  contractual  deliverable  . 

d.  Architecture  evaluations  will  be  (are  being)  adopted  as  the 

means  for  identifying  architecture  risks  early  in  the  acquisition 
life  cycle  . 

e.  The  ATAM  will  be  (is  being)  used  to  conduct  architecture  eval¬ 

uations  on  major  upgrades  or  new  systems  the  program  office  or 
its  stakeholders  will  be  responsible  for . 

f.  On  new  contract  starts,  a  QAW  will  be  (is  being)  proactively 

specified  in  the  RFP/contract  as  part  of  the  up-front  acquisition 
planning  process  . 

g.  On  new  contract  starts,  an  ATAM  will  be  (is  being)  proactively 

specified  in  the  RFP/contract  as  part  of  the  up-front  acquisition 
planning  process  . 

h.  PMO  and  supplier  will  be  (are)  negotiating  changes  to  require¬ 
ments  based  on  a  better  understanding  of  program  constraints . 

i.  Suppliers  will  be  (are  being)  contractually  required  to  produce  a 

formal  architecture  risk  mitigation  plan  . 

j.  Program  office  and  or  supplier  personnel  will  be  (are  being) 

appropriately  trained  so  they  can  be  part  of  the  ATAM  Evalua¬ 
tion  Team  . 

k.  Additional  program  office  personnel  will  take  (are  taking)  train¬ 
ing  to  become  SEI-certified  ATAM  Evaluators  . 

l.  Other  (please  identify): 
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m.  Please  describe  why  you  adopted  or  plan  to  adopt  the  practices  noted.  What  was  the  realized 
benefit?  What  factors  enabled  or  contributed  to  the  success  of  performing  these  practices? 


n.  Please  describe  why  you  did  not  adopt  the  practices  you  noted.  Even  though  not  performed, 
is  there  a  perceived  benefit  to  doing  so?  What  were  the  obstacles  that  hindered  you  from 
doing  so? 
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Do  Not  Plan 
to  Adopt 


IV.  Overall  Impact 

In  view  of  the  entire  engagement,  the  following  sections  address  the  impact  of  the 
ATAM/QAW  in  terms  of  the  effects  on  up-front  cost  and  schedule  and  longer  term  value  in 
terms  of  cost  savings  (avoidance),  schedule,  quality,  and  capability  improvements. 


1.  Cost,  Schedule,  and  Quality  Impact 

If  the  ATAM/QAW  had  not  been  conducted,  some  level  of  effort  would  have  been  spent  on 
eliciting  and  specifying  the  nonfunctional  (e.g.,  quality  attribute)  requirements  and  evaluating 
the  software  design  (e.g.,  peer  reviews,  walkthroughs)  using  other  means. 


Please  answer  the  following  two  questions  from  the  perspective 
of  using  your  other  means  to  achieve  the  same  level  of  quality 
that  you  realized  from  conducting  the  ATAM/QAW: 

a.  Compared  to  the  likely  effort  that  would  have  been  expended, 

the  ATAM/QAW  was  less,  the  same,  or  more  effort  . 

b.  Compared  to  the  likely  cost  that  would  have  been  incurred,  the  □  □  □ 

ATAM/QAW  was  less,  the  same,  or  more  cost  . 

Please  answer  the  following  two  questions  from  the  perspective 
of  using  your  other  means  as  you  traditionally  do: 

c.  Comparing  the  quality  of  the  results  of  other  means  used  for  □  □  □ 

eliciting  and  specifying  the  nonfunctional  requirements,  the 

ATAM/QAW  produced  results  (e.g.,  quality  attribute  scenarios) 
that  were  less,  the  same,  or  more  quality . 

d.  Comparing  the  quality  of  the  results  of  other  means  used  for  □  □  □ 

evaluating  software  design,  the  ATAM  produced  results  (e.g., 

quality  attribute  scenarios,  architecture  documentation,  and  arc¬ 
hitectural  risks)  that  were  less,  the  same,  or  more  quality . 

e.  What  data  is  being  collected  to  support  your  answers  noted  above?  Can  you  provide  a  more 
quantitative  response  to  the  above  questions? 

f.  Comments: 
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2.  Long-Term  Value  of  the  ATAM/QAW  Results 

Based  on  your  experience  please  provide  your  best  reasoned  estimate  of  the  value  your  pro¬ 
gram  received  from  the  ATAM/QAW  experience  and  results  (e.g.,  better  and  earlier  identifi¬ 
cation  of  quality  attribute  requirements,  documentation  of  architecture  design  decisions,  and 
discovery  of  architecture  risks): 
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a.  System  Quality  Improved — positive  impact  on  sys- 
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tem  acceptance  and  usability  through  ability  to  pro- 

vide  the  affected  qualities . 

b.  Program  Schedule  Performance  Improved — positive 

□ 

□ 

□ 

□ 

impact  on  controlling  schedule  (e.g.,  minimizing 
scale  of  the  added  schedule  delay  that  would  other¬ 
wise  have  been  required  downstream  had  the  risks 

not  been  discovered  early) . 

c.  Program  Cost  Performance  Improved — positive  im-  □  □  □  □ 

pact  on  controlling  cost  (e.g.,  minimizing  scale  of 
the  cost  associated  with  the  rework  effort  that  would 
otherwise  have  been  required  downstream  had  the 
risks  not  been  discovered  early)  . 


d.  Warfighter  Effectiveness  Improved — positive  impact  □  □  □  □ 

on  mission  goals  through  ability  to  provide  the  af¬ 
fected  capabilities  and  qualities . 


e.  What  data  is  being  collected  to  support  your  answers  noted  above?  Can  you  provide  a  more 
quantitative  response  to  the  above  questions? 


f.  Comments: 
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3.  Summary 


a.  What,  in  your  opinion,  was  the  overall  impact  of  the  ATAM/QAW  on  the  architecture? 


b.  What,  in  your  opinion,  was  the  overall  value  of  the  ATAM/QAW?  Do  you  feel  the  benefit  ex¬ 
ceeded  its  cost? 


c.  Are  there  any  other  comments  that  you  wish  to  share? 
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Appendix  D  Raw  Responses  Regarding  Impact 


Captured  in  this  appendix  are  the  raw  responses  to  the  last  three  questions  of  the  survey,  which 

were  open-ended  questions  seeking  to  elicit  a  qualitative  sense  of  overall  value.  The  responses  are 

provided  for  completeness. 

1 .  What,  in  your  opinion,  was  the  overall  impact  of  the  ATAM/QAW  on  the  architecture? 

2.  What,  in  your  opinion,  was  the  overall  value  of  the  ATAM/QAW?  Do  you  feel  the  benefit 
exceeded  its  cost? 

3.  Are  there  any  other  comments  that  you  wish  to  share? 

Program  1 

•  On  an  existing  architecture  across  multiple  platforms  across  multiple  services,  the  ATAM 
process  is  not  effective,  due  to  cost  and  schedule  constraints.  Individual  platform  changes  to 
architecture  [are]  costly  and  schedule  intensive,  with  severe  impacts  on  training  and  the  lo¬ 
gistics  tail. 

.  No. 

•  The  ATAM  was  performed  to  determine  if  it  could  be  incorporated  into  the  development 
effort.  Unfortunately,  due  to  the  breath  of  impact  of  the  architecture,  the  results  were  not 
amenable  to  implementation  except  at  great  cost  and  time.  ATAMs  should  be  conducted  at 
the  initiation  of  programs  where  architectural  changes  can  be  readily  integrated  into  the  pro¬ 
gram  development.  The  [program]  community  is  aware,  and  has  been  for  some  time,  of 
changes  to  the  architecture  which  would  improve  the  product.  Finding  the  means  to  imple¬ 
ment  these  changes  without  bankrupting  the  efforts  is  the  difficulty. 

Program  2 

•  The  ATAM  and  QAW  had  a  positive  impact  to  the  DoDAF  architecture  construction,  but  no 
impact  on  the  contracted  efforts. 

•  The  benefit  of  the  ATAM  was  equal  to  its  cost.  The  ATAM  was  conducted  too  soon  in  the 
system  life  cycle. 

Program  3 

•  The  contractor  team  was  already  doing  very  good  work  on  the  software  architecture  prior  to 
the  ATAM.  I  think  the  peer  review  was  positive  in  that  it  challenged  some  of  the  developers’ 
thinking  and  helped  everyone  better  understand  the  capabilities  we  were  trying  to  achieve. 
There  is  no  doubt  that  the  project  office  exited  the  activity  with  a  better  understanding  of  the 
requirements  and  how  we  intended  to  satisfy  them. 

•  The  benefit  was  significant  and  exceeded  the  costs.  The  software  has  demonstrated  great 
utility  and  flexibility.  What  is  more,  the  system  has  transitioned  to  a  critical,  high  profile  hel¬ 
icopter  upgrade  program. 

•  The  SEI  did  a  very  good  job  presiding  over,  facilitating,  and  leading  our  ATAM.  They  have 
very  highly  qualified  technical  experts  and  have  laid  out  a  process  that  proved  to  be  very 
beneficial  to  us. 


36  |  CMU/SEI-2009-SR-007 


Program  4 

•  For  the  ATAM  portion  of  the  process,  which  is  the  scope  of  this  response,  the  impact  was 
minimal. 

•  I  do  not  think  the  ATAM  itself  gave  us  a  net  positive  return  on  investment.  It  did  help  with 
stakeholder  communication,  but  ungrounded  QA  inputs  and  functionality  expectations  in  the 
form  of  scenarios  from  stakeholders  required  extra  non-value  added  work.  Architecture 
Evaluations  accordingly  to  the  modified  [program]  process  have  been  very  valuable. 

•  This  particular  QAW  was  very  large  (~80  participants).  The  architects  were  experienced  and 
had  detailed  quality  attribute  definitions  from  past  programs  that  were  adapted  for  use  on 
[the  system].  The  brainstorming  method  of  identifying  QAs  in  the  QAW  was  therefore  unne¬ 
cessary  and  of  lower  quality  than  the  detailed  analysis  which  formed  the  basis  of  the  prior 
work.  Many  of  the  scenarios  postulated  by  participants  were  unrealistic  and  unnecessary  to 
meet  vague  and  emerging  program  requirements.  So  there  was  a  strong  sense  of  “unreality” 
in  the  scenarios,  and  the  QA  inputs  from  stakeholders  lacked  depth. 

•  From  a  technical  perspective  then,  the  QAW  served  more  as  a  review  of  the  proposed  QAs 
than  a  means  of  defining  them.  The  SEI  facilitators  were  experienced  and  conducted  the 
QAW  well.  A  tailoring  of  the  QAW  was  coordinated  with  them  beforehand  to  meet  the  un¬ 
precedented  scale  of  this  QAW,  but  then  a  ruling  from  the  SEI  came  down  that  it  could  not 
be  called  a  “QAW”  unless  it  followed  the  standard  process  verbatim.  This  was  unfortunate 
and  led  to  wasted  effort  in  the  workshop. 

•  Asa  result  of  the  above,  the  value  of  the  QAW  was  socio-political.  It  got  the  stakeholders  to 
agree  on  the  QAs,  even  though  the  scenarios  did  not  survive.  It  allowed  the  program  to  state 
that  it  had  followed  a  standard  process  in  doing  so.  The  resulting  QAs  were  established  in  a 
stable  manner  and  have  formed  a  good  basis  for  the  [program] -specific  architecture  evalua¬ 
tion  process. 

Program  5 

•  As  the  ATAM/QAW  was  exercised  late  in  [the  system’s]  development  process,  it  verified 
and  validated  the  quality  of  the  architecture  to  a  larger  set  of  representatives  within  the  user 
space.  Secondarily,  the  ATAM/QAW  identified  a  consistent  set  of  risk  themes  for  tracking 
by  the  government  and  suppliers  teams. 

•  Although  I  do  not  have  a  specific  cost  value  to  tie  to  the  ATAM,  the  ATAM/QAW  was 
beneficial  to  strengthen  the  quality  attribute  identification  and  architecture  development 
processes  and  I  believe  the  benefits  far  exceeded  the  costs.  The  ATAM/QAW  process  is  es¬ 
sential  early  in  the  development  of  complex,  software-intensive  systems  for  consistent  stake¬ 
holder  (user,  supplier,  management)  requirement  (functional  and  nonfunctional)  focus,  de¬ 
sign,  implementation,  and  acceptance.  Continued  architecture  maintenance  and  quality 
attribute  alignment/sustaimnent  is  also  viewed  as  necessary  component  of  the  greater  soft¬ 
ware  life-cycle  management  process. 

•  Overall,  the  ATAM/QAW  was  beneficial  to  [the  program]  as  it  provided  an  independent, 
rigorous  means  of  assessing  the  architecture  from  a  use  case,  quality  attribute  perspective  in 
front  of  a  multitude  of  stakeholders.  This  assessment  provided  valuable,  high-quality  insight 
into  the  risks  and  non-risks  associated  with  the  architecture  in  meeting  the  stakeholder  needs 
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and  requirements.  Finally,  the  ATAM/QAW  process  and  report  provided  both  the  govern¬ 
ment  program  office  and  the  contracted  supplier  the  confidence  and  stakeholder  support  to 
address  follow-on  user  requirements  that  far  exceeded  the  original  capabilities  as  imple¬ 
mented  within  the  architecture. 

•  The  ATAM/QAW  is  highly  recommended  for  new  software-intensive  projects  committed  to 
engaging  their  stakeholder  community,  practicing  sound  software  architecture  and  develop¬ 
mental  processes,  and  exercising  a  meaningful  risk  identification  and  management  process. 
Even  for  ongoing  projects  the  ATAM/QAW  process  is  very  beneficial  for  an  independent  as¬ 
sessment  of  architecture  products  and  a  structured  process  for  identifying  architecture-related 
risks  and  non-risks. 

Program  6 

•  The  AT  AM  did  not  have  as  much  impact  because  most  of  the  work  was  done  up  front  in  our 
case  by  the  architecture  team  between  [the  conducting  of  the]  QAW  and  AT  AM.  The  two 
QAWs  we  held  had  significant  impact  that  drove  the  architecture  with  respect  to  level  of 
embedded-ness,  abstraction  layers,  and  modularity  from  what  we  had  before. 

•  Gathering  all  the  stakeholders  in  one  room  in  itself  was  a  huge  benefit.  QAW,  AT  AM  did 
not  exceed  its  cost,  if  anything  we  should  have  done  more  of  them  to  continually  reconfirm 
nonfunctional  requirements,  update  the  architecture,  and  get  buy-in  to  the  architecture. 

•  Product  line  requires  wholesale  changes  in  PMO  ways  of  doing  business  from  technical, 
business,  and  requirements  (TRADOC  Capabilities  Manager  [TCM])  perspective  first;  oth¬ 
erwise,  there  is  no  hope  a  defense  contractor  will  never  change.  Another  comment  is  there 
needs  to  be  competition  and  multiple  sources  for  [software]  SW  modules  up  front;  otherwise, 
1  feel  we  are  not  fully  leveraging  a  documented,  modular  architecture  if  we  keep  going  to 
one  source  and  expect  to  meet  our  strategic  goals:  better,  faster,  cheaper. 

Program  7 

•  The  overall  impact  of  the  ATAM/QAW  on  the  system  architecture  was  positive,  ft  was  posi¬ 
tive  primarily  because  the  process  requires  the  supplier  architecture  team  to  explain  to  evalu¬ 
ators/stakeholders  how  the  proposed  system  architecture  would  respond  in  various  scenarios. 

•  No,  the  system  architecture  was  changed  where  appropriate  to  address  the  resultant  risk 
themes.  The  insertion  of  an  AT  AM-like  process  earlier  in  the  development  did  find  risks  ear¬ 
ly  and  thus  would  have  reduced  long-term  cost. 

•  In  [the  respondent’s]  opinion,  a  system  AT  AM-like  process  could  have  a  significantly  posi¬ 
tive  impact  to  the  construction  of  the  Department  of  Defense  Architecture  Framework  (Do- 
DAF)  architectural  artifacts  developed  by  the  Program  Managers  and  the  TRADOC  System 
Managers  (TSM).  The  current  AT  AM  process  is  for  software  and  obtains  its  strength  via  the 
originators’  understanding  of  the  applicability  of  software  patterns.  At  the  higher  level 
DODAF  architectural  abstractions,  knowledge  domain  for  such  enterprise  patterns  is  a  dark 
and  empty  void.  We  had  no  comments  on  patterns. 
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Program  8 

•  QAWs  and  AT  AMs  had  a  positive  impact  on  decision  making  with  risk  themes  and  tradeoffs 
that  could  be  analyzed  to  influence  future  architecture  vision. 

•  Structured,  repeatable  process  to  understand  risk  in  architectural  approaches 

•  Both  ATAMs  and  QAWs  provided  exceptional  overall  value  for  achieving  Army  goals  with 
joint  programs. 

Program  9 

•  QAWs  and  ATAMs  had  a  positive  impact  on  decision  making  with  risk  themes  and  tradeoffs 
that  could  be  analyzed  to  influence  future  architecture  vision. 

•  Structured,  repeatable  process  to  understand  risk  in  architectural  approaches 

•  Both  ATAMs  and  QAWs  provided  exceptional  overall  value  for  achieving  Army  goals  with 
joint  programs. 

Program  10 

•  The  QAW  and  AT  AM  were  intended  to  understand  the  design  decisions  made  regarding  the 
architecture  of  an  existing  system  and  explore  growth  scenarios  to  evolve  that  architecture  to 
a  future  system.  The  application  (via  contracts)  of  the  QAW  realized  quality  attribute  re¬ 
quirements  and  ATAM-based  evaluations  will  impact  the  shape  of  the  architecture  of  future 
systems. 

•  The  overall  value  of  the  ATAM/QAW  is  that  several  specific  activities  were  initiated  that 
will  affect  future  architectures.  These  include 

ATAM-identified  risks  and  mitigation  suggestions  were  incorporated  into  the  program 
risk  database. 

An  ATAM-based  Software  Architecture  Evaluation  Plan  was  written  and  incorporated 
into  an  acquisition  program. 

The  QAW  process  (whose  proponent  was  initially  the  software  group)  was  subsequently 
adopted  by  System  Engineering,  and  a  second  QAW  (post-ATAM  and  not  facilitated  by 
SEI)  was  held. 

QAW  results  were  utilized  to  explore  use  cases  and  as  a  basis  for  development  of  non¬ 
functional  System  Specification  requirements. 

Benchmarking  of  key  technologies  and  products  identified  [by  means  of  the]  ATAM  as 
risks  were  initiated  or  expanded. 

QAW  use  case  operational  scenarios  were  examined  for  substantiation  in  the  System  Ar¬ 
chitecture. 

Program  11 

•  Existing  documentation  was  improved  and  documentation  thereafter  was  of  better  quality. 
With  a  better  understanding  of  the  requirements,  further  refinement  of  the  architecture  went 
smoother. 

•  The  largest  impact  was  improved  understanding  and  communications.  This  was  particularly 
true  since  we  had  just  merged  two  competing  contracts  into  a  single  partnering  contract.  This 
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resulted  in  a  variety  of  process  improvements  that  have  been  resulting  in  better  development 
and  a  better  understanding,  resulting  in  cost  savings  through  less  rework. 

•  Full  implementation  of  the  ATAM/QAW  process  will  not  happen  until  it  is  added  to  the 
processes  required  by  DoD  5000. lS  The  same  holds  true  of  other  ASSIP  efforts.  PMs  do  what 
PMs  are  funded  to  do,  and  PMs  are  funded  to  meet  requirements. 

•  Contracts  are  follow-on  extensions.  No  new  programs  [are]  coming  online. 

Program  12 

•  QAWs  and  AT  AMs  had  a  positive  impact  on  decision  making  with  risk  themes  and  tradeoffs 
that  could  be  analyzed  to  influence  future  architecture  vision. 

•  Structured,  repeatable  process  to  understand  risk  in  architectural  approaches 

•  Both  ATAMs  and  QAWs  provided  exceptional  overall  value  for  achieving  Army  goals  with 
joint  programs. 


DoD  5000.01  details  the  policies  that  govern  the  U.S.  DoD  acquisition  system;  DoD  5000.02  details  the  man¬ 
agement  framework  that  implements  those  policies. 
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